You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by bjlovegithub <gi...@git.apache.org> on 2016/10/13 10:49:01 UTC
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
GitHub user bjlovegithub opened a pull request:
https://github.com/apache/flink/pull/2629
[FLINK-4391] Provide support for asynchronous operations over streams
PR for [FLIP 12](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673) - Implementation for Async I/O in FLINK.
Basically, the newly added operator supports async operation while streaming. By implementing AsyncFunction, we can easily do i/o access, like fetching data from HBase, in async way.
Please refer to AsyncIOExample.java about how to use it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bjlovegithub/flink master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2629.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2629
----
commit f352d0a8b879164baa8e4922f63ad8dde13c7c9b
Author: yushi.wxg <yu...@taobao.com>
Date: 2016-10-12T06:31:43Z
Add async wait operator
commit 139902396e1c30d88671ac67a54afea145ab7651
Author: yushi.wxg <yu...@taobao.com>
Date: 2016-10-13T10:13:32Z
1. add an example job 2. fix a bug in state serialization in async wait operator; 3. move broadcast barrier after snapshot operator states
commit fc41eedcd48d2dd7d940e744dec86a51b24e4ac5
Author: yushi.wxg <yu...@taobao.com>
Date: 2016-10-13T10:25:36Z
update IT case
commit 9855f071f04b6efcfc4e10a9f6ef1c38ae264637
Author: yushi.wxg <yu...@taobao.com>
Date: 2016-10-13T10:28:39Z
adjust the whitespace in IT
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88676639
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(sleepFactor*20, TimeUnit.MILLISECONDS);
--- End diff --
If `awaitTermination` fails, then we should call `executorService.shutdownNow()`.
I would introduce a shutdown timeout which can be specified by the user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
No this case can also happen if you don't have an input for a really long time, e.g. a completely consumed Kafka topic to which no data is written anymore. In this case you would still like to output the pending elements asap without adding the latency of waiting for a new element to arrive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86759643
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
+ Thread.sleep(1000);
--- End diff --
Alright, this makes sense. With the addition that we can eagerly terminate the `run` method in case of an error, we could also get rid of the `sleep`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86524130
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -540,15 +540,12 @@ private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws
synchronized (lock) {
if (isRunning) {
+ checkpointState(checkpointMetaData);
- // Since both state checkpointing and downstream barrier emission occurs in this
- // lock scope, they are an atomic operation regardless of the order in which they occur.
- // Given this, we immediately emit the checkpoint barriers, so the downstream operators
- // can start their checkpoint work as soon as possible
+ // broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
- checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp());
-
- checkpointState(checkpointMetaData);
+ checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
+ );
--- End diff --
I think the `ReentrantReadWriteLock` could work. However, I'm not so sure whether the higher costs of this lock compared to a mutual exclusion lock we're currently using is worth the change. I fear that we're optimising here for the case where you have a long chain of `AsyncWaitOperators`. Instead we could simply disallow chaining for these operators. Then every chain would have at most 2 write threads (main and `Emitter`) competing for the lock. Thus, I would vote for using the existing mutual exclusion lock instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
For exactly once processing guarantees, when `AsyncCollectorBuffer.getStreamElementsInBuffer` has finished, `isCheckpointing` will be set to true, while will idle `Emitter Thread` while checking `nonthingToDo()`. So, once snapshotting in `StreamTask` is triggered, all `EmitterThread` have stopped there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86506096
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
--- End diff --
`collectorToStreamElement` is removed. Actually, `AsyncCollector` acts like an internal object, and should not be overwritten. So the behavior for `AsyncCollector` in `HashMap` and `IdentityHashMap` are the same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
The combination of `UNORDERED` with event time is not strictly meaningless. The only thing which one has to do respect is that only those elements in the queue from the beginning to the first watermark element are allowed to be processed in an `UNORDERED` fashion. Only they have been emitted, the watermark can be emitted. After the watermark has been emitted, the next batch of stream records can be processed `UNORDERED`. An example:
Given `queue = s1, s2, s3, w1, s4, w2, s5, s6, w3` we can process `s1, s2` `s3` `UNORDERED`. After they have been emitted, `w1` has to be emitted. And only after the emission of `w1`, we can start emitting `s4`. After `w2` we can process `s5` and `s6` `UNORDERED`. And so on...
Concerning the exactly once processing guarantees. The problem is the following with the current implementation. When you call `AsyncWaitOperator.snapshotState` it will call `AsyncCollectorBuffer.getStreamElementsInBuffer` which will block the `Emitter` thread. However, after `getStreamElementsInBuffer` has completed, the `Emitter` thread is again allowed to emit new elements, right? Thus, this can happen before the `StreamTask` has actually send the checkpoint barrier to downstream operators because it is not guarded by the checkpoint lock. So you might have an element `x1` contained in the checkpoint of the `AsyncWaitOperator` and in one of the downstream operators. That's the reason why you only have at-least once processing guarantees.
I fear that the hand-tailored solution for the `AsyncWaitOperator` in the `StreamTask.performCheckpoint` method won't solve the problems with the exactly once processing guarantees because of the afore-mentioned problems. I don't see another way to solve the problem at the moment other than using the checkpoint lock. Have you measured the performance penalty for multiple chained `AsyncWaitOperators` when using the checkpoint lock?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
hi @tillrohrmann @mproch I have updated the PR, including a wrapper class for `RuntimeContext` specialized for `RichAsyncFunction`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88721238
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/api/StreamingOperatorsITCase.java ---
@@ -195,6 +202,70 @@ public Integer map(NonSerializable value) throws Exception {
env.execute();
}
+ @Test
+ public void testAsyncWaitOperator() throws Exception {
+ final int numElements = 10;
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ DataStream<Tuple2<Integer, NonSerializable>> input = env.addSource(new NonSerializableTupleSource(numElements)).setParallelism(1);
+
+ AsyncFunction<Tuple2<Integer, NonSerializable>, Integer> function = new RichAsyncFunction<Tuple2<Integer, NonSerializable>, Integer>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(numElements);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ }
+
+ @Override
+ public void asyncInvoke(final Tuple2<Integer, NonSerializable> input,
+ final AsyncCollector<Tuple2<Integer, NonSerializable>, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * 1000);
+ try {
+ Thread.sleep(sleep);
+ List<Integer> ret = new ArrayList<>();
+ ret.add(input.f0+input.f0);
+ collector.collect(ret);
+ }
+ catch (InterruptedException e) {
+ collector.collect(new ArrayList<Integer>(0));
+ }
+ }
+ });
+ }
+ };
+
+ DataStream<Integer> orderedResult = AsyncDataStream.orderedWait(input, function, 2).setParallelism(1);
+ orderedResult.writeAsText(resultPath1, FileSystem.WriteMode.OVERWRITE).setParallelism(1);
+
+ DataStream<Integer> unorderedResult = AsyncDataStream.unorderedWait(input, function, 2).setParallelism(1);
+ unorderedResult.writeAsText(resultPath2, FileSystem.WriteMode.OVERWRITE);
--- End diff --
Writing to disk can fail on travis. Thus, it is highly recommended rewriting these tests doing so that they store their results in memory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89424283
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private final AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ Preconditions.checkNotNull(buffer, "Reference to AsyncCollectorBuffer should not be null");
+
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer, boolean isDone) {
+ this(buffer);
+ this.isDone = isDone;
--- End diff --
Actually, they work as a stub here. I think we can optimize those `AsyncCollector`s for `Watermark` and `LatencyMarker` out while optimizing the code in `nothingToDo()`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
Ah, I think now I understand. Upon checkpointing you disable the output emission. The emission can only be reactivated when adding a `StreamRecord`, a `Watermark` or a `LatencyMarker` to the `AsyncCollectorBuffer`. And this cannot happen as long as the checkpointing isn't completed, because it holds the checkpoint lock.
But isn't this a problem in the following case: What if we have some elements pending in the `queue` when doing the checkpoint. After the checkpoint has completed, there won't be any other `StreamRecords`, `Watermarks` (because we haven't activated them) and `LatencyMarkers` coming on the input. This would imply that the pending elements in the `AsyncCollectorBuffer` will never be emitted, won't they?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86312266
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
--- End diff --
`AsyncCollector` does not override `toString`. What information do you want to display here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86332972
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
--- End diff --
Not strictly enforced, but most of Flink's code puts every parameter in a single line double indented if one breaks the parameter list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88719571
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
+
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setOutputMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ @VisibleForTesting
+ public void setEmitFlag(boolean emitFlag) {
+ this.emitFlag = emitFlag;
+ }
+
+ @VisibleForTesting
+ public void setInStreamElementSerializerForTest(StreamElementSerializer<IN> serializer) {
+ this.inStreamElementSerializerForTest = serializer;
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ if (this.inStreamElementSerializerForTest != null) {
+ this.inStreamElementSerializer = this.inStreamElementSerializerForTest;
+ }
+ else {
+ this.inStreamElementSerializer = new StreamElementSerializer(this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader()));
+ }
+
+ this.collector = new TimestampedCollector<>(output);
+
+ this.checkpointLock = containingTask.getCheckpointLock();
+
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, output, collector, this.checkpointLock, this);
+
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.recoveredStreamElements != null) {
+ for (StreamElement element : this.recoveredStreamElements.get()) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ }
+ else if (element.isWatermark()) {
+ processWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ processLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown record type: "+element.getClass());
+ }
+ }
+ this.recoveredStreamElements = null;
+ }
+
+
+ if (emitFlag) {
+ buffer.startEmitterThread();
+ }
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.addStreamRecord(element);
+
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.addWatermark(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.addLatencyMarker(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ ListState<StreamElement> partitionableState =
+ getOperatorStateBackend().getOperatorState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer));
+ partitionableState.clear();
+
+ Collection<StreamElement> elements = buffer.getStreamElementsInBuffer();
--- End diff --
Can't we simply return an `Iterable` over the `StreamElements` in the `buffer`? Then we don't have to create a list where we first copy all elements from the buffer and then return this list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86333894
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
--- End diff --
I think the `inputsFromState` can be of type `Collection`. Maybe we could rename this field into `unprocessedStreamElements` or `recoveredStreamElements`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
Hi @tillrohrmann . I have updated the PR. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86508318
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
--- End diff --
Yep, I will try to throw an exception while compiling the job graph for this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88701658
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
--- End diff --
It does not throw an `Exception`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89291235
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
--- End diff --
You mean starting the `emitter thread` while constructing `AsyncCollectorBuffer`? Since we have to get a `AsyncCollectorBuffer` ready to put inputs from previous state while `open()` the operator. After all inputs are in the buffer, `emitter thread` then can start working.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86492478
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86312736
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
--- End diff --
This could maybe be an `IOException`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86200419
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
--- End diff --
I know that we don't have a strict code style in Flink, but could we maybe put every parameter double indented in a single line when breaking a parameter list? So that could be
```
void foobar(
int a,
float b) {
barfoo();
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86511814
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88869631
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector. Try to emit results prior to the oldest watermark
+ * in the queue.
+ * <p>
+ * For example, assume the queue layout is:
+ * Entry(ac1, record1) -> Entry(ac2, record2) -> Entry(ac3, watermark1) -> Entry(ac4, record3).
+ * and both of ac2 and ac3 have finished. For unordered-mode, ac1 and ac2 are prior to watermark1,
+ * so ac2 will be emitted. Since ac1 is not ready yet, ac3 have to wait until ac1 is done.
+ */
+ private void unorderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean hasUnfinishedCollector = false;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ StreamElement element = entry.getValue();
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (element.isWatermark() && hasUnfinishedCollector) {
+ break;
+ }
+
+ if (collector.isDone() == false) {
+ hasUnfinishedCollector = true;
+
+ continue;
+ }
+
+ try {
+ output(collector, element);
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors prior to the oldest water mark.
+ * or
+ * The first element in the queue is Watermark or LatencyMarker.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
--- End diff --
Can't we avoid this method by simply having a volatile flag which is set by the `markCollectorCompleted` if the first collector was completed (in case of ordered processing), or if we added marker as the first element? In unordered processing mode, we would additionally have to store the positions of the watermarks in the queue. Then we wouldn't have to iterate over the queue over and over again if nothing happens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88866589
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
+
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setOutputMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ @VisibleForTesting
+ public void setEmitFlag(boolean emitFlag) {
+ this.emitFlag = emitFlag;
+ }
+
+ @VisibleForTesting
+ public void setInStreamElementSerializerForTest(StreamElementSerializer<IN> serializer) {
+ this.inStreamElementSerializerForTest = serializer;
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ if (this.inStreamElementSerializerForTest != null) {
+ this.inStreamElementSerializer = this.inStreamElementSerializerForTest;
+ }
+ else {
+ this.inStreamElementSerializer = new StreamElementSerializer(this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader()));
+ }
+
+ this.collector = new TimestampedCollector<>(output);
+
+ this.checkpointLock = containingTask.getCheckpointLock();
+
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, output, collector, this.checkpointLock, this);
+
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.recoveredStreamElements != null) {
+ for (StreamElement element : this.recoveredStreamElements.get()) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ }
+ else if (element.isWatermark()) {
+ processWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ processLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown record type: "+element.getClass());
+ }
+ }
+ this.recoveredStreamElements = null;
+ }
+
+
+ if (emitFlag) {
+ buffer.startEmitterThread();
+ }
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.addStreamRecord(element);
+
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.addWatermark(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.addLatencyMarker(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ ListState<StreamElement> partitionableState =
+ getOperatorStateBackend().getOperatorState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer));
+ partitionableState.clear();
+
+ Collection<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ for (StreamElement element : elements) {
+ partitionableState.add(element);
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ recoveredStreamElements =
+ context.getManagedOperatorStateStore().getOperatorState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer));
+
+ }
+
+ @Override
+ public void close() throws Exception {
+ // for test only, we do not have to wait until buffer is empty.
+ if (!emitFlag) {
+ return;
+ }
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
--- End diff --
I think we should still stop the emitter thread even if an exception is thrown by `buffer.waitEmpty` otherwise there might be a resource leak.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88865406
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperatorTest.java ---
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.testutils.OneShotLatch;
+import org.apache.flink.runtime.checkpoint.CheckpointMetaData;
+import org.apache.flink.runtime.checkpoint.SubtaskState;
+import org.apache.flink.runtime.operators.testutils.MockInputSplitProvider;
+import org.apache.flink.runtime.state.TaskStateHandles;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTask;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTaskTestHarness;
+import org.apache.flink.streaming.runtime.tasks.StreamMockEnvironment;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.TestHarnessUtil;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Tests for {@link AsyncWaitOperator}. These test that:
+ *
+ * <ul>
+ * <li>Process StreamRecords and Watermarks in ORDERED mode</li>
+ * <li>Process StreamRecords and Watermarks in UNORDERED mode</li>
+ * <li>Snapshot state and restore state</li>
+ * </ul>
+ */
+public class AsyncWaitOperatorTest {
+
+ public static class MyAsyncFunction extends RichAsyncFunction<Integer, Integer> {
+ transient final int SLEEP_FACTOR = 100;
+ transient final int THREAD_POOL_SIZE = 10;
--- End diff --
why are these values `transient`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89285857
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
--- End diff --
Placing `AsyncCollectorBuffer` in `functions` package seems not very nice. I create a buffer sub-package under `async` and place it here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86505374
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
--- End diff --
Cool! I have thought about `LinkedHashMap`, this class meets our requirements perfectly. The following two maps, `collectorToStreamElement` and the `collectorToQueue`, are also can be deleted, so does `SimpleLinkedList` :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86198219
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
--- End diff --
Why do we want to minimize the latency in the AsyncIOExample?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86321156
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
+ Thread.sleep(1000);
+ }
+ else {
+ while (nothingToDo())
--- End diff --
Flink usually always uses curly brackets also for if/while statements whose body has only a single line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r92436971
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/async/RichAsyncFunction.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.async;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RichFunction;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.streaming.api.functions.async.collector.AsyncCollector;
+
+/**
+ * Rich variant of the {@link AsyncFunction}. As a {@link RichFunction}, it gives access to the
+ * {@link org.apache.flink.api.common.functions.RuntimeContext} and provides setup and teardown methods:
+ * {@link RichFunction#open(org.apache.flink.configuration.Configuration)} and
+ * {@link RichFunction#close()}.
+ *
+ * <p>
+ * {@link RichAsyncFunction#getRuntimeContext()} and {@link RichAsyncFunction#getRuntimeContext()} are
+ * not supported because the key may get changed while accessing states in the working thread.
+ *
+ * @param <IN> The type of the input elements.
+ * @param <OUT> The type of the returned elements.
+ */
+
+@PublicEvolving
+public abstract class RichAsyncFunction<IN, OUT> extends AbstractRichFunction
+ implements AsyncFunction<IN, OUT> {
+
+ @Override
+ public abstract void asyncInvoke(IN input, AsyncCollector<OUT> collector) throws Exception;
+
+ @Override
+ public RuntimeContext getRuntimeContext() {
--- End diff --
Maybe we could wrap the `RuntimeContext` which will throw an `UnsupportedOperationException` if we try to access state. Everything else will be forwarded to the underlying `RuntimeContext`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88676904
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.ListCheckpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, ListCheckpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public List<Integer> snapshotState(long checkpointId, long timestamp) throws Exception {
+ return Collections.singletonList(start);
+ }
+
+ @Override
+ public void restoreState(List<Integer> state) throws Exception {
+ for (Integer i : state)
+ this.start = i;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+ private static void printUsage() {
+ System.out.println("To customize example, use: AsyncIOExample [--fsStatePath <path to fs state>] " +
+ "[--checkpointMode <exactly_once or at_least_once>] [--maxCount <max number of input from source>] " +
+ "[--sleepFactor <interval to sleep for each stream element>] [--failRatio <possibility to throw exception>] " +
+ "[--waitMode <ordered or unordered>] [--waitOperatorParallelism <parallelism for async wait operator>] " +
+ "[--eventType <EventTime or IngestionTime>]");
+ }
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ printUsage();
+
+ // parse parameters
+ final ParameterTool params = ParameterTool.fromArgs(args);
+
+ // check the configuration for the job
+ final String statePath = params.getRequired("fsStatePath");
+ final String cpMode = params.get("checkpointMode", "exactly_once");
+ final int maxCount = params.getInt("maxCount", 100000);
+ final int sleepFactor = params.getInt("sleepFactor", 100);
+ final float failRatio = params.getFloat("failRatio", 0.001f);
+ final String mode = params.get("waitMode", "ordered");
+ final int taskNum = params.getInt("waitOperatorParallelism", 1);
+ final String timeType = params.get("eventType", "EventTime");
+
+ System.out.println("Job configuration\n"
+ +"\tFS state path="+statePath+"\n"
+ +"\tCheckpoint mode="+cpMode+"\n"
+ +"\tMax count of input from source="+maxCount+"\n"
+ +"\tSleep factor="+sleepFactor+"\n"
+ +"\tFail ratio="+failRatio+"\n"
+ +"\tWaiting mode="+mode+"\n"
+ +"\tParallelism for async wait operator="+taskNum+"\n"
+ +"\tEvent type="+timeType);
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of an single integer
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+ transient Random random;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
+ random = new Random();
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(sleepFactor*20, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, String> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (random.nextFloat() * sleepFactor);
+ try {
+ Thread.sleep(sleep);
+ List<String> ret = new ArrayList<>();
+ ret.add("key-"+(input%10));
--- End diff --
This could also be a singleton list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86197886
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(sleepFactor*20, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, String> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * sleepFactor);
+ try {
+ Thread.sleep(sleep);
+ List<String> ret = new ArrayList<>();
+ ret.add("key-"+(input%10));
+ if (new Random().nextFloat() > failRatio) {
--- End diff --
Shouldn't it be `random.nextFloat() < failRatio`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86517044
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
+ stream.write(input.size());
+
+ for (StreamElement element : input) {
+ if (element.isRecord()) {
+ stream.write(1);
+
+ StreamRecord<IN> record = element.asRecord();
+
+ outputSerializer.clear();
+ inTypeSerializer.serialize(record.getValue(), outputSerializer);
+ outputSerializer.writeLong(record.getTimestamp());
+ outputSerializer.writeBoolean(record.hasTimestamp());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else if (element.isWatermark()) {
+ stream.write(0);
+
+ Watermark watermark = element.asWatermark();
+
+ outputSerializer.clear();
+ outputSerializer.writeLong(watermark.getTimestamp());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else if (element.isLatencyMarker()) {
+ stream.write(2);
+
+ LatencyMarker marker = element.asLatencyMarker();
+
+ outputSerializer.clear();
+ outputSerializer.writeLong(marker.getMarkedTime());
+ outputSerializer.writeInt(marker.getVertexID());
+ outputSerializer.writeInt(marker.getSubtaskIndex());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else {
+ throw new IOException("Unknown element type: "+element);
+ }
+ }
+ }
+
+ private List<StreamElement> deserializeStreamElements(FSDataInputStream stream) throws IOException {
+ DataInputViewStreamWrapper wrapper = new DataInputViewStreamWrapper(stream);
+
+ int size = wrapper.read();
+
+ List<StreamElement> ret = new ArrayList<>(size);
+ for (int i = 0; i < size; ++i) {
+ int flag = wrapper.read();
+
+ if (flag == 1) {
--- End diff --
Reply to the serialize and deserialize part. StreamElementSerializer is so cool :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89293393
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
--- End diff --
You mean the `StreamElement` in the parameter list?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88865801
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperatorTest.java ---
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.testutils.OneShotLatch;
+import org.apache.flink.runtime.checkpoint.CheckpointMetaData;
+import org.apache.flink.runtime.checkpoint.SubtaskState;
+import org.apache.flink.runtime.operators.testutils.MockInputSplitProvider;
+import org.apache.flink.runtime.state.TaskStateHandles;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTask;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTaskTestHarness;
+import org.apache.flink.streaming.runtime.tasks.StreamMockEnvironment;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.TestHarnessUtil;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Tests for {@link AsyncWaitOperator}. These test that:
+ *
+ * <ul>
+ * <li>Process StreamRecords and Watermarks in ORDERED mode</li>
+ * <li>Process StreamRecords and Watermarks in UNORDERED mode</li>
+ * <li>Snapshot state and restore state</li>
+ * </ul>
+ */
+public class AsyncWaitOperatorTest {
+
+ public static class MyAsyncFunction extends RichAsyncFunction<Integer, Integer> {
+ transient final int SLEEP_FACTOR = 100;
+ transient final int THREAD_POOL_SIZE = 10;
+ transient ExecutorService executorService;
+ transient Random random;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
+ random = new Random();
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(SLEEP_FACTOR*THREAD_POOL_SIZE, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (random.nextFloat() * SLEEP_FACTOR);
+ try {
+ Thread.sleep(sleep);
+ List<Integer> ret = new ArrayList<>();
+ ret.add(input*2);
+ collector.collect(ret);
+ }
+ catch (InterruptedException e) {
+ }
+ }
+ });
+ }
+ }
+
+ @Test
+ public void testWaterMarkOrdered() throws Exception {
+ testWithWatermark(AsyncDataStream.OutputMode.ORDERED);
+ }
+
+ @Test
+ public void testWaterMarkUnordered() throws Exception {
+ testWithWatermark(AsyncDataStream.OutputMode.UNORDERED);
+ }
+
+ private void testWithWatermark(AsyncDataStream.OutputMode mode) throws Exception {
+ final AsyncWaitOperator<Integer, Integer> operator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ operator.setOutputMode(mode);
+ operator.setBufferSize(6);
+ operator.setInStreamElementSerializerForTest(new StreamElementSerializer<>(IntSerializer.INSTANCE));
+
+ final OneInputStreamOperatorTestHarness<Integer, Integer> testHarness =
+ new OneInputStreamOperatorTestHarness<>(operator);
+
+ final long initialTime = 0L;
+ final ConcurrentLinkedQueue<Object> expectedOutput = new ConcurrentLinkedQueue<>();
+
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(1, initialTime + 1));
+ testHarness.processElement(new StreamRecord<>(2, initialTime + 2));
+ testHarness.processWatermark(new Watermark(initialTime + 2));
+ testHarness.processElement(new StreamRecord<>(3, initialTime + 3));
+
+ // wait until all async collectors in the buffer have been emitted out.
+ synchronized (testHarness.getCheckpointLock()) {
+ testHarness.close();
+ }
+
+ expectedOutput.add(new StreamRecord<>(2, initialTime + 1));
+ expectedOutput.add(new StreamRecord<>(4, initialTime + 2));
+ expectedOutput.add(new Watermark(initialTime + 2));
+ expectedOutput.add(new StreamRecord<>(6, initialTime + 3));
+
+ if (AsyncDataStream.OutputMode.ORDERED == mode) {
+ TestHarnessUtil.assertOutputEquals("Output with watermark was not correct.", expectedOutput, testHarness.getOutput());
+ }
+ else {
+ Assert.assertTrue(expectedOutput.size() == 4);
+ Assert.assertTrue(expectedOutput.toArray(new StreamElement[4])[2].asWatermark().getTimestamp() == (initialTime+2));
--- End diff --
Can't we make this check a little bit more exhaustive?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86335385
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
+ stream.write(input.size());
+
+ for (StreamElement element : input) {
+ if (element.isRecord()) {
+ stream.write(1);
--- End diff --
If using the `StreamElementSerializer`, then we don't have to take care of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86709933
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -540,15 +540,12 @@ private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws
synchronized (lock) {
if (isRunning) {
+ checkpointState(checkpointMetaData);
- // Since both state checkpointing and downstream barrier emission occurs in this
- // lock scope, they are an atomic operation regardless of the order in which they occur.
- // Given this, we immediately emit the checkpoint barriers, so the downstream operators
- // can start their checkpoint work as soon as possible
+ // broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
- checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp());
-
- checkpointState(checkpointMetaData);
+ checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
+ );
--- End diff --
Can you share the test program and test result? I write an sample program to test the performance to acquire and release lock with `Object` / `ReentrantLock` / `ReentrantReadWriteLock`, finding that for one thread to use the lock, they have similar results. But referring to multiple thread cases, `ReentrantLock` and ` ReentrantReadWriteLock` excels `Object` lock.
Here is my benchmark program: [link](https://github.com/bjlovegithub/JavaLockTest/blob/master/src/LockTest.java)
And this is the sampled results run on my laptop: [stat](https://github.com/bjlovegithub/JavaLockTest/blob/master/stat/LockTest_result.data)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88857262
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
--- End diff --
Can't we simply call `AsyncWaitOperator#setup` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86196655
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
--- End diff --
Can we harden the parameter parsing a little bit and print a help message in case the user specifies the wrong parameters?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86320859
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
--- End diff --
we could write this as `return finishedCollectors.isEmpty();`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86356070
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
--- End diff --
I think that `OutputMode.UNORDERED` won't work with event time processing. The watermarks can easily overtake elements with a smaller timestamp.
Either way, we don't allow the `OutputMode.UNORDERED` when event time was chosen, or we handle watermarks always in order. This means that the watermark is only emitted after all preceding elements have been emitted (potentially in an unordered fashion).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86347448
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
--- End diff --
Shouldn't we and the test fail in case of an `Exception`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88858547
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+
+ buffer.addWatermark(new Watermark(0l));
+ buffer.addLatencyMarker(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(buffer.getQueue().size(), 2);
+
+ Iterator<Map.Entry<AsyncCollector<Integer, Integer>, StreamElement>> iterator =
+ buffer.getQueue().entrySet().iterator();
+ Watermark watermark = iterator.next().getValue().asWatermark();
+ Assert.assertEquals(watermark.getTimestamp(), 0l);
+
+ LatencyMarker latencyMarker = iterator.next().getValue().asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp()) {
+ sb.append(record.getTimestamp() + ",");
+ }
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs) {
+ sb.append(i).append(",");
+ }
+ return sb.toString();
+ }
+
+ public Collection<Long> getOutputs() {
+ return outputs;
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ private void work(final boolean throwExcept) throws Exception {
+ final int ASYNC_COLLECTOR_NUM = 7;
+
+ final int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ final int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*100;
--- End diff --
The sleeps are too long. Would be great to decrease them. say to max 100ms or so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83380179
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
+ }
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, String> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * sleepFactor);
+ try {
+ Thread.sleep(sleep);
+ List<String> ret = new ArrayList<>();
+ ret.add("key-"+(input%10));
+ if (new Random().nextFloat() > failRatio) {
+ collector.collect(new Exception("wahahahaha..."));
--- End diff --
This is my favorite line :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r92315209
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/async/RichAsyncFunction.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.async;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RichFunction;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.streaming.api.functions.async.collector.AsyncCollector;
+
+/**
+ * Rich variant of the {@link AsyncFunction}. As a {@link RichFunction}, it gives access to the
+ * {@link org.apache.flink.api.common.functions.RuntimeContext} and provides setup and teardown methods:
+ * {@link RichFunction#open(org.apache.flink.configuration.Configuration)} and
+ * {@link RichFunction#close()}.
+ *
+ * <p>
+ * {@link RichAsyncFunction#getRuntimeContext()} and {@link RichAsyncFunction#getRuntimeContext()} are
+ * not supported because the key may get changed while accessing states in the working thread.
+ *
+ * @param <IN> The type of the input elements.
+ * @param <OUT> The type of the returned elements.
+ */
+
+@PublicEvolving
+public abstract class RichAsyncFunction<IN, OUT> extends AbstractRichFunction
+ implements AsyncFunction<IN, OUT> {
+
+ @Override
+ public abstract void asyncInvoke(IN input, AsyncCollector<OUT> collector) throws Exception;
+
+ @Override
+ public RuntimeContext getRuntimeContext() {
--- End diff --
Personally, I prefer the second option... It is handy for the user codes, even though the limitations present.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r92314677
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/async/RichAsyncFunction.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.async;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RichFunction;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.streaming.api.functions.async.collector.AsyncCollector;
+
+/**
+ * Rich variant of the {@link AsyncFunction}. As a {@link RichFunction}, it gives access to the
+ * {@link org.apache.flink.api.common.functions.RuntimeContext} and provides setup and teardown methods:
+ * {@link RichFunction#open(org.apache.flink.configuration.Configuration)} and
+ * {@link RichFunction#close()}.
+ *
+ * <p>
+ * {@link RichAsyncFunction#getRuntimeContext()} and {@link RichAsyncFunction#getRuntimeContext()} are
+ * not supported because the key may get changed while accessing states in the working thread.
+ *
+ * @param <IN> The type of the input elements.
+ * @param <OUT> The type of the returned elements.
+ */
+
+@PublicEvolving
+public abstract class RichAsyncFunction<IN, OUT> extends AbstractRichFunction
+ implements AsyncFunction<IN, OUT> {
+
+ @Override
+ public abstract void asyncInvoke(IN input, AsyncCollector<OUT> collector) throws Exception;
+
+ @Override
+ public RuntimeContext getRuntimeContext() {
--- End diff --
Hi @mproch, In our use case in the production env, we provide some metrics in the `AsyncWaitOperator`, like the number of elements in the `AsyncCollectorBuffer` etc. Maybe the `AsyncWaitOperator` provides those basic statistics metrics.
The `RichAsyncFunction` is useful for its `open()` and `close()` methods, where the async clients can be initialized and freed.
I think there are two options:
1. Remove `RichAsyncFunction`, since `AsyncWaitOperator` currently only supports non-keyed stream.
2. Keep `RichAsyncFunction` but it is different to use `RichAsyncFunction.getRuntimeContext()`...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83380500
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
+ }
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, String> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * sleepFactor);
+ try {
+ Thread.sleep(sleep);
+ List<String> ret = new ArrayList<>();
+ ret.add("key-"+(input%10));
+ if (new Random().nextFloat() > failRatio) {
+ collector.collect(new Exception("wahahahaha..."));
+ }
+ else {
+ collector.collect(ret);
+ }
+ }
+ catch (InterruptedException e) {
+ collector.collect(new ArrayList<String>(0));
+ }
+ }
+ });
+ }
+
+ private void readObject(java.io.ObjectInputStream in)
--- End diff --
How about initializing the executor service in `open(Configuration)`? That would be easier to understand, for all the example readers that do not know all the details of Java Serialization.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86492216
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(sleepFactor*20, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, String> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * sleepFactor);
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86762586
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -540,15 +540,12 @@ private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws
synchronized (lock) {
if (isRunning) {
+ checkpointState(checkpointMetaData);
- // Since both state checkpointing and downstream barrier emission occurs in this
- // lock scope, they are an atomic operation regardless of the order in which they occur.
- // Given this, we immediately emit the checkpoint barriers, so the downstream operators
- // can start their checkpoint work as soon as possible
+ // broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
- checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp());
-
- checkpointState(checkpointMetaData);
+ checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
+ );
--- End diff --
I think the important performance characteristic is not how the locks perform when there is only a single thread trying to acquire the lock but if there is some kind of lock contention by multiple threads. If you change your benchmark so that you have multiple threads running at the same time, you will see that the `ReentrantLock` and the `ReentrantReadWriteLock` are much more expensive than the `Object`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88858824
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+
+ buffer.addWatermark(new Watermark(0l));
+ buffer.addLatencyMarker(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(buffer.getQueue().size(), 2);
+
+ Iterator<Map.Entry<AsyncCollector<Integer, Integer>, StreamElement>> iterator =
+ buffer.getQueue().entrySet().iterator();
+ Watermark watermark = iterator.next().getValue().asWatermark();
+ Assert.assertEquals(watermark.getTimestamp(), 0l);
+
+ LatencyMarker latencyMarker = iterator.next().getValue().asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp()) {
+ sb.append(record.getTimestamp() + ",");
+ }
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs) {
+ sb.append(i).append(",");
+ }
+ return sb.toString();
+ }
+
+ public Collection<Long> getOutputs() {
+ return outputs;
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ private void work(final boolean throwExcept) throws Exception {
+ final int ASYNC_COLLECTOR_NUM = 7;
+
+ final int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ final int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*100;
+
+ if (i == 3) {
+ synchronized (lock) {
+ buffer.addWatermark(new Watermark(333l));
+ }
+ }
+ else if (i == 6) {
+ synchronized (lock) {
+ buffer.addLatencyMarker(new LatencyMarker(111L, 0, 0));
+ }
+ }
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+
+ AsyncCollector tmp;
+ synchronized (lock) {
+ tmp = buffer.addStreamRecord(record);
+ }
+
+ final AsyncCollector collector = tmp;
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+
+ if (throwExcept) {
+ collector.collect(new Exception("wahahahaha..."));
+ }
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (InterruptedException e) {
+ }
+ }
+ });
+ }
+ }
+
+ synchronized (lock) {
+ buffer.waitEmpty();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, output, collector, lock, operator);
+ buffer.startEmitterThread();
+
+ work(false);
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+
+ Assert.assertEquals(((FakedOutput)output).getResult(), "0,1,2,333,4,5,111,");
+ Assert.assertEquals(((FakedOutput)output).getTimestamp(), "0,1,4,16,25,");
+ }
+
+ private String sort(String s) {
+ String[] arr = s.split(",");
+ int[] ints = new int[arr.length];
+ for (int i = 0; i < arr.length; ++i) {
+ ints[i] = Integer.valueOf(arr[i]);
+ }
+ Arrays.sort(ints);
+ return Arrays.toString(ints);
+ }
+
+ @Test
+ public void testUnorderedBuffer() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+ buffer.startEmitterThread();
+
+ work(false);
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+
+ Assert.assertEquals(((FakedOutput)output).getOutputs().toArray()[3], 333L);
+ Assert.assertEquals(sort(((FakedOutput)output).getResult()), "[0, 1, 2, 4, 5, 111, 333]");
+ Assert.assertEquals(sort(((FakedOutput)output).getTimestamp()), "[0, 1, 4, 16, 25]");
+ }
+
+ @Test(expected = IOException.class)
+ public void testBufferWithException() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+ buffer.startEmitterThread();
+
+ work(true);
+
+ buffer.waitEmpty();
--- End diff --
I think you have to check a little bit better whether the thrown exception is actually our expected test exception. What if its another `IOException` originating from somewhere else.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88680681
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
--- End diff --
Maybe the `AsyncCollectorBuffer` belongs into the `functions` package as the `AsyncFunction`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86319267
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
--- End diff --
Can we use `IdentityHashMaps` here? The `HashMap` behaviour with `AsyncCollector` as key boils basically down to a `IdentityHashMap`. Using this map here, will make this semantic property more specific and guards against sub classes of `AsyncCollector` which overwrite the `equals` implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86308486
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
--- End diff --
maybe we could rename `maxSize` into `bufferSize`. This makes it clearer what the parameter is used for.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86311749
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
--- End diff --
`Preconditions.checkNotNull` missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89294937
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
--- End diff --
Actually it does not have to be wrapped io `IOException`. I will fixed later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86367216
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
--- End diff --
Can't we pass the `collector` and the `output` to the constructor of the `AsyncCollectorBuffer`? Then we could make these fields final as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86914170
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
--- End diff --
Ah, that makes sense. No matter what condition is, the `waitEmpty()` will be signaled. So it is not necessary to `processError()` in `Emitter Thread` repeatedly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86507829
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86527698
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/api/StreamingOperatorsITCase.java ---
@@ -195,6 +202,70 @@ public Integer map(NonSerializable value) throws Exception {
env.execute();
}
+ @Test
+ public void testAsyncWaitOperator() throws Exception {
+ final int numElements = 10;
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ DataStream<Tuple2<Integer, NonSerializable>> input = env.addSource(new NonSerializableTupleSource(numElements)).setParallelism(1);
+
+ AsyncFunction<Tuple2<Integer, NonSerializable>, Integer> function = new RichAsyncFunction<Tuple2<Integer, NonSerializable>, Integer>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(numElements);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ }
+
+ @Override
+ public void asyncInvoke(final Tuple2<Integer, NonSerializable> input,
+ final AsyncCollector<Tuple2<Integer, NonSerializable>, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * 1000);
+ try {
+ Thread.sleep(sleep);
+ List<Integer> ret = new ArrayList<>();
+ ret.add(input.f0+input.f0);
+ collector.collect(ret);
+ }
+ catch (InterruptedException e) {
+ collector.collect(new ArrayList<Integer>(0));
+ }
+ }
+ });
+ }
+ };
+
+ DataStream<Integer> orderedResult = AsyncDataStream.orderedWait(input, function, 2).setParallelism(1);
+ orderedResult.writeAsText(resultPath1, FileSystem.WriteMode.OVERWRITE).setParallelism(1);
+
+ DataStream<Integer> unorderedResult = AsyncDataStream.unorderedWait(input, function, 2).setParallelism(1);
+ unorderedResult.writeAsText(resultPath2, FileSystem.WriteMode.OVERWRITE);
--- End diff --
You don't have to use the `compareResultsByLinesInMemory` to check the results. Then you don't have to write it on disk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86493149
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setMode(mode);
+
+ OneInputTransformation<IN, OUT> resultTransform = new OneInputTransformation<>(
+ in.getTransformation(),
+ "async wait operator",
+ operator,
+ outTypeInfo,
+ in.getExecutionEnvironment().getParallelism());
+
+ SingleOutputStreamOperator<OUT> returnStream =
+ new SingleOutputStreamOperator<>(in.getExecutionEnvironment(), resultTransform);
+
+ returnStream.getExecutionEnvironment().addOperator(resultTransform);
+
+ return returnStream;
+ }
+
+ /**
+ * Add an AsyncWaitOperator. The order of output stream records may be reordered.
+ *
+ * @param in Input {@link DataStream}
+ * @param func AsyncFunction
+ * @bufSize The max number of async i/o operation that can be triggered
+ * @return A new {@link SingleOutputStreamOperator}.
+ */
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize) {
+ return addOperator(in, func, bufSize, OutputMode.UNORDERED);
+ }
+
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86484584
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
--- End diff --
I will use ParameterTool and print out the usage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88867342
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperatorTest.java ---
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.testutils.OneShotLatch;
+import org.apache.flink.runtime.checkpoint.CheckpointMetaData;
+import org.apache.flink.runtime.checkpoint.SubtaskState;
+import org.apache.flink.runtime.operators.testutils.MockInputSplitProvider;
+import org.apache.flink.runtime.state.TaskStateHandles;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTask;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTaskTestHarness;
+import org.apache.flink.streaming.runtime.tasks.StreamMockEnvironment;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.TestHarnessUtil;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Tests for {@link AsyncWaitOperator}. These test that:
+ *
+ * <ul>
+ * <li>Process StreamRecords and Watermarks in ORDERED mode</li>
+ * <li>Process StreamRecords and Watermarks in UNORDERED mode</li>
+ * <li>Snapshot state and restore state</li>
+ * </ul>
+ */
+public class AsyncWaitOperatorTest {
+
+ public static class MyAsyncFunction extends RichAsyncFunction<Integer, Integer> {
+ transient final int SLEEP_FACTOR = 100;
+ transient final int THREAD_POOL_SIZE = 10;
+ transient ExecutorService executorService;
+ transient Random random;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
+ random = new Random();
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(SLEEP_FACTOR*THREAD_POOL_SIZE, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (random.nextFloat() * SLEEP_FACTOR);
+ try {
+ Thread.sleep(sleep);
+ List<Integer> ret = new ArrayList<>();
+ ret.add(input*2);
+ collector.collect(ret);
+ }
+ catch (InterruptedException e) {
+ }
+ }
+ });
+ }
+ }
+
+ @Test
+ public void testWaterMarkOrdered() throws Exception {
+ testWithWatermark(AsyncDataStream.OutputMode.ORDERED);
+ }
+
+ @Test
+ public void testWaterMarkUnordered() throws Exception {
+ testWithWatermark(AsyncDataStream.OutputMode.UNORDERED);
+ }
+
+ private void testWithWatermark(AsyncDataStream.OutputMode mode) throws Exception {
+ final AsyncWaitOperator<Integer, Integer> operator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ operator.setOutputMode(mode);
+ operator.setBufferSize(6);
+ operator.setInStreamElementSerializerForTest(new StreamElementSerializer<>(IntSerializer.INSTANCE));
+
+ final OneInputStreamOperatorTestHarness<Integer, Integer> testHarness =
+ new OneInputStreamOperatorTestHarness<>(operator);
+
+ final long initialTime = 0L;
+ final ConcurrentLinkedQueue<Object> expectedOutput = new ConcurrentLinkedQueue<>();
+
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(1, initialTime + 1));
+ testHarness.processElement(new StreamRecord<>(2, initialTime + 2));
+ testHarness.processWatermark(new Watermark(initialTime + 2));
+ testHarness.processElement(new StreamRecord<>(3, initialTime + 3));
+
+ // wait until all async collectors in the buffer have been emitted out.
+ synchronized (testHarness.getCheckpointLock()) {
+ testHarness.close();
+ }
+
+ expectedOutput.add(new StreamRecord<>(2, initialTime + 1));
+ expectedOutput.add(new StreamRecord<>(4, initialTime + 2));
+ expectedOutput.add(new Watermark(initialTime + 2));
+ expectedOutput.add(new StreamRecord<>(6, initialTime + 3));
+
+ if (AsyncDataStream.OutputMode.ORDERED == mode) {
+ TestHarnessUtil.assertOutputEquals("Output with watermark was not correct.", expectedOutput, testHarness.getOutput());
+ }
+ else {
+ Assert.assertTrue(expectedOutput.size() == 4);
+ Assert.assertTrue(expectedOutput.toArray(new StreamElement[4])[2].asWatermark().getTimestamp() == (initialTime+2));
+ }
+ }
+
+ @Test
+ public void testOrdered() throws Exception {
+ testRun(AsyncDataStream.OutputMode.ORDERED);
+ }
+
+ @Test
+ public void testUnordered() throws Exception {
+ testRun(AsyncDataStream.OutputMode.UNORDERED);
+ }
+
+ private void testRun(AsyncDataStream.OutputMode mode) throws Exception {
+ final OneInputStreamTask<Integer, Integer> task = new OneInputStreamTask<>();
+ final OneInputStreamTaskTestHarness<Integer, Integer> testHarness =
+ new OneInputStreamTaskTestHarness<>(task, 1, 1, BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO);
+
+ final AsyncWaitOperator<Integer, Integer> operator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ operator.setOutputMode(mode);
+ operator.setBufferSize(6);
+
+ final StreamConfig streamConfig = testHarness.getStreamConfig();
+ streamConfig.setStreamOperator(operator);
+
+ testHarness.invoke();
+ testHarness.waitForTaskRunning();
+
+ final long initialTime = 0L;
+ final ConcurrentLinkedQueue<Object> expectedOutput = new ConcurrentLinkedQueue<Object>();
+
+ testHarness.processElement(new StreamRecord<>(1, initialTime + 1));
+ testHarness.processElement(new StreamRecord<>(2, initialTime + 2));
+ testHarness.processElement(new StreamRecord<>(3, initialTime + 3));
+ testHarness.processElement(new StreamRecord<>(4, initialTime + 4));
+ testHarness.processElement(new StreamRecord<>(5, initialTime + 5));
+ testHarness.processElement(new StreamRecord<>(6, initialTime + 6));
+ testHarness.processElement(new StreamRecord<>(7, initialTime + 7));
+ testHarness.processElement(new StreamRecord<>(8, initialTime + 8));
+
+ expectedOutput.add(new StreamRecord<>(2, initialTime + 1));
+ expectedOutput.add(new StreamRecord<>(4, initialTime + 2));
+ expectedOutput.add(new StreamRecord<>(6, initialTime + 3));
+ expectedOutput.add(new StreamRecord<>(8, initialTime + 4));
+ expectedOutput.add(new StreamRecord<>(10, initialTime + 5));
+ expectedOutput.add(new StreamRecord<>(12, initialTime + 6));
+ expectedOutput.add(new StreamRecord<>(14, initialTime + 7));
+ expectedOutput.add(new StreamRecord<>(16, initialTime + 8));
+
+ testHarness.waitForInputProcessing();
+
+ testHarness.endInput();
+
+ testHarness.waitForTaskCompletion();
+
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ TestHarnessUtil.assertOutputEquals("ORDERED Output was not correct.", expectedOutput, testHarness.getOutput());
+ }
+ else {
+ TestHarnessUtil.assertOutputEqualsSorted(
+ "UNORDERED Output was not correct.",
+ expectedOutput,
+ testHarness.getOutput(),
+ new Comparator<Object>() {
+ @Override
+ public int compare(Object o1, Object o2) {
+ Integer val1 = ((StreamRecord<Integer>)o1).getValue();
+ Integer val2 = ((StreamRecord<Integer>)o2).getValue();
+ if (val1 > val2) {
+ return 1;
+ }
+ else if (val1 == val2) {
+ return 0;
+ }
+ else {
+ return -1;
+ }
+ }
+ });
+ }
+ }
+
+ @Test
+ public void testStateSnapshotAndRestore() throws Exception {
+ final OneInputStreamTask<Integer, Integer> task = new OneInputStreamTask<>();
+ final OneInputStreamTaskTestHarness<Integer, Integer> testHarness =
+ new OneInputStreamTaskTestHarness<>(task, 1, 1, BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO);
+
+ final AsyncWaitOperator<Integer, Integer> operator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ operator.setOutputMode(AsyncDataStream.OutputMode.ORDERED);
+ operator.setBufferSize(6);
+ // disable starting emitter thread, so that all inputs will be kept in the buffer, which will be
+ // snapshotted into operator state
+ operator.setEmitFlag(false);
+
+ final StreamConfig streamConfig = testHarness.getStreamConfig();
+ streamConfig.setStreamOperator(operator);
+
+ final AcknowledgeStreamMockEnvironment env = new AcknowledgeStreamMockEnvironment(
+ testHarness.jobConfig,
+ testHarness.taskConfig,
+ testHarness.getExecutionConfig(),
+ testHarness.memorySize,
+ new MockInputSplitProvider(),
+ testHarness.bufferSize);
+
+ testHarness.invoke(env);
+ testHarness.waitForTaskRunning();
+
+ final long initialTime = 0L;
+
+ testHarness.processElement(new StreamRecord<>(1, initialTime + 1));
+ testHarness.processElement(new StreamRecord<>(2, initialTime + 2));
+ testHarness.processElement(new StreamRecord<>(3, initialTime + 3));
+ testHarness.processElement(new StreamRecord<>(4, initialTime + 4));
+
+ testHarness.waitForInputProcessing();
+
+ final long checkpointId = 1L;
+ final long checkpointTimestamp = 1L;
+
+ final CheckpointMetaData checkpointMetaData = new CheckpointMetaData(checkpointId, checkpointTimestamp);
+
+ while(!task.triggerCheckpoint(checkpointMetaData));
+
+ env.getCheckpointLatch().await();
+
+ assertEquals(checkpointId, env.getCheckpointId());
+
+ testHarness.endInput();
+ testHarness.waitForTaskCompletion();
+
+ env.getCheckpointStateHandles();
+
+ // set the operator state from previous attempt into the restored one
+ final OneInputStreamTask<Integer, Integer> restoredTask = new OneInputStreamTask<>();
+ restoredTask.setInitialState(new TaskStateHandles(env.getCheckpointStateHandles()));
+
+ final OneInputStreamTaskTestHarness<Integer, Integer> restoredTaskHarness =
+ new OneInputStreamTaskTestHarness<>(restoredTask, BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO);
+
+ final AsyncWaitOperator<Integer, Integer> restoredOperator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ restoredOperator.setOutputMode(AsyncDataStream.OutputMode.ORDERED);
+ restoredOperator.setBufferSize(6);
+
+ restoredTaskHarness.getStreamConfig().setStreamOperator(restoredOperator);
+
+ restoredTaskHarness.invoke();
+ restoredTaskHarness.waitForTaskRunning();
+
+ restoredTaskHarness.processElement(new StreamRecord<>(5, initialTime + 5));
+ restoredTaskHarness.processElement(new StreamRecord<>(6, initialTime + 6));
+ restoredTaskHarness.processElement(new StreamRecord<>(7, initialTime + 7));
+ restoredTaskHarness.processElement(new StreamRecord<>(8, initialTime + 8));
+
+ restoredTaskHarness.endInput();
+ restoredTaskHarness.waitForTaskCompletion();
+
+ ConcurrentLinkedQueue<Object> expectedOutput = new ConcurrentLinkedQueue<>();
+ expectedOutput.add(new StreamRecord<>(2, initialTime + 1));
+ expectedOutput.add(new StreamRecord<>(4, initialTime + 2));
+ expectedOutput.add(new StreamRecord<>(6, initialTime + 3));
+ expectedOutput.add(new StreamRecord<>(8, initialTime + 4));
+ expectedOutput.add(new StreamRecord<>(10, initialTime + 5));
+ expectedOutput.add(new StreamRecord<>(12, initialTime + 6));
+ expectedOutput.add(new StreamRecord<>(14, initialTime + 7));
+ expectedOutput.add(new StreamRecord<>(16, initialTime + 8));
+
+ TestHarnessUtil.assertOutputEquals("StateAndRestored Test Output was not correct.", expectedOutput, restoredTaskHarness.getOutput());
+ }
--- End diff --
What about checkpointing tests where some elements have been processed and others not? That way we can see that the processed elements are properly removed from the internal state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88701684
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
--- End diff --
Does not throw an `Exception`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86493542
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private final AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer, boolean isDone) {
+ this(buffer);
+ this.isDone = isDone;
+ }
+
+ /**
+ * Set result
+ * @param result A list of results.
+ */
+ public void collect(List<OUT> result) {
--- End diff --
Yes, only for one time. I will update the code doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88715005
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector. Try to emit results prior to the oldest watermark
+ * in the queue.
+ * <p>
+ * For example, assume the queue layout is:
+ * Entry(ac1, record1) -> Entry(ac2, record2) -> Entry(ac3, watermark1) -> Entry(ac4, record3).
+ * and both of ac2 and ac3 have finished. For unordered-mode, ac1 and ac2 are prior to watermark1,
+ * so ac2 will be emitted. Since ac1 is not ready yet, ac3 have to wait until ac1 is done.
+ */
+ private void unorderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean hasUnfinishedCollector = false;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ StreamElement element = entry.getValue();
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (element.isWatermark() && hasUnfinishedCollector) {
+ break;
+ }
+
+ if (collector.isDone() == false) {
+ hasUnfinishedCollector = true;
+
+ continue;
+ }
+
+ try {
+ output(collector, element);
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors prior to the oldest water mark.
+ * or
+ * The first element in the queue is Watermark or LatencyMarker.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ return true;
+ }
+
+ // get the first AsyncCollector and StreamElement in the queue.
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> firstEntry = queue.entrySet().iterator().next();
+ StreamElement element = firstEntry.getValue();
+ AsyncCollector<IN, OUT> collector = firstEntry.getKey();
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (element.isWatermark() || element.isLatencyMarker()) {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ return !collector.isDone();
+ }
+ else {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean noFinishedCollector = true;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ if (entry.getKey().isDone()) {
+ noFinishedCollector = false;
+ break;
+ }
+
+ if (entry.getValue().isWatermark()) {
+ break;
+ }
+ }
+
+ return noFinishedCollector;
+ }
+ }
+
+ private void processFinishedAsyncCollector() throws IOException {
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ orderedProcess();
+ } else {
+ unorderedProcess();
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ synchronized (lock) {
+ try {
+ while (nothingToDo()) {
+ lock.wait();
+ }
+ }
+ catch(InterruptedException e) {
+ break;
+ }
+
+ try {
+ processFinishedAsyncCollector();
+ }
+ catch (IOException e) {
+ queue.clear();
+ error = e;
+
+ lock.notifyAll();
+
+ break;
--- End diff --
`running = false`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83573944
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer, boolean isDone) {
+ this(buffer);
+ this.isDone = isDone;
+ }
+
+ /**
+ * Set result
+ * @param result A list of results.
+ */
+ public void collect(List<OUT> result) {
+ this.result = result;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+ * Set error
+ * @param error A Throwable object.
+ */
+ public void collect(Throwable error) {
+ this.error = error;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+ * Get result. Throw RuntimeException while encountering an error.
+ *
+ * @return A List of result.
+ * @throws RuntimeException RuntimeException wrapping errors from user codes.
+ */
+ public List<OUT> getResult() throws RuntimeException {
--- End diff --
That makes me clarify about how to use RuntimeException ;D
I prefer using IOException, since the error is from I/O process. I will change RuntimeException into IOException in the codes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88714941
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
--- End diff --
Why do you wrap the exception by the way?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88867234
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/api/StreamingOperatorsITCase.java ---
@@ -195,6 +202,70 @@ public Integer map(NonSerializable value) throws Exception {
env.execute();
}
+ @Test
+ public void testAsyncWaitOperator() throws Exception {
+ final int numElements = 10;
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ DataStream<Tuple2<Integer, NonSerializable>> input = env.addSource(new NonSerializableTupleSource(numElements)).setParallelism(1);
+
+ AsyncFunction<Tuple2<Integer, NonSerializable>, Integer> function = new RichAsyncFunction<Tuple2<Integer, NonSerializable>, Integer>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(numElements);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ }
+
+ @Override
+ public void asyncInvoke(final Tuple2<Integer, NonSerializable> input,
+ final AsyncCollector<Tuple2<Integer, NonSerializable>, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * 100);
+ try {
+ Thread.sleep(sleep);
--- End diff --
Can we avoid sleeps? These make the tests just slow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86333293
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
--- End diff --
Can `input` be of type `Collection`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
Hi @bjlovegithub, just wanted check whether I can help you with anything to complete this PR and get it merged. If that's the case, then let me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88675892
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
--- End diff --
An example should not be a test cast. Thus, I think we should remove this divergent behaviour.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86312565
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
--- End diff --
Shouldn't we check this in the `AsyncCollector`? We could then either fail there or store the exception as the result of the `AsyncCollector`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88707755
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
--- End diff --
Isn't this an `IOException`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88712042
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
--- End diff --
I think the try block should go outside of the loop since you exit the loop in the catch block.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2629
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86201630
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setMode(mode);
+
+ OneInputTransformation<IN, OUT> resultTransform = new OneInputTransformation<>(
+ in.getTransformation(),
+ "async wait operator",
+ operator,
+ outTypeInfo,
+ in.getExecutionEnvironment().getParallelism());
+
+ SingleOutputStreamOperator<OUT> returnStream =
+ new SingleOutputStreamOperator<>(in.getExecutionEnvironment(), resultTransform);
+
+ returnStream.getExecutionEnvironment().addOperator(resultTransform);
+
+ return returnStream;
+ }
+
+ /**
+ * Add an AsyncWaitOperator. The order of output stream records may be reordered.
+ *
+ * @param in Input {@link DataStream}
+ * @param func AsyncFunction
+ * @bufSize The max number of async i/o operation that can be triggered
+ * @return A new {@link SingleOutputStreamOperator}.
+ */
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize) {
+ return addOperator(in, func, bufSize, OutputMode.UNORDERED);
+ }
+
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
--- End diff --
JavaDocs missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86518429
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
--- End diff --
It is not necessary, I will remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86366712
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
--- End diff --
We don't actually have to create an `AsyncCollector` for a watermark or a latency marker. Thus, I think it would be better if we stored in our `LinkedHashSet` an `AsyncStreamElement` which is abstract. The `AsyncStreamElement` gives you access to the `StreamElement`. The `StreamRecords` implementation of `AsyncStreamElement` also implements `AsyncCollector`, which allows it to be passed into the `AsyncFunction`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88718820
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
+
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setOutputMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ @VisibleForTesting
+ public void setEmitFlag(boolean emitFlag) {
+ this.emitFlag = emitFlag;
+ }
+
+ @VisibleForTesting
+ public void setInStreamElementSerializerForTest(StreamElementSerializer<IN> serializer) {
+ this.inStreamElementSerializerForTest = serializer;
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ if (this.inStreamElementSerializerForTest != null) {
+ this.inStreamElementSerializer = this.inStreamElementSerializerForTest;
+ }
+ else {
+ this.inStreamElementSerializer = new StreamElementSerializer(this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader()));
+ }
+
+ this.collector = new TimestampedCollector<>(output);
+
+ this.checkpointLock = containingTask.getCheckpointLock();
+
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, output, collector, this.checkpointLock, this);
+
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.recoveredStreamElements != null) {
+ for (StreamElement element : this.recoveredStreamElements.get()) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ }
+ else if (element.isWatermark()) {
+ processWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ processLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown record type: "+element.getClass());
+ }
+ }
+ this.recoveredStreamElements = null;
+ }
+
+
+ if (emitFlag) {
+ buffer.startEmitterThread();
+ }
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.addStreamRecord(element);
+
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.addWatermark(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.addLatencyMarker(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ ListState<StreamElement> partitionableState =
+ getOperatorStateBackend().getOperatorState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer));
+ partitionableState.clear();
+
+ Collection<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ for (StreamElement element : elements) {
+ partitionableState.add(element);
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ recoveredStreamElements =
+ context.getManagedOperatorStateStore().getOperatorState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer));
+
+ }
+
+ @Override
+ public void close() throws Exception {
+ // for test only, we do not have to wait until buffer is empty.
+ if (!emitFlag) {
+ return;
+ }
--- End diff --
This is not good. Test code should not be part of the production code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86706002
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
+ Thread.sleep(1000);
--- End diff --
Sorry for the delayed response.
Currently, the main thread will detect errors while:
1. `AsyncWaitOperator.processElement()`, thrown by `AsyncCollectorBuffer.addStreamRecord()`.
2. `AsyncWaitOperator.processWatermark()` and `AsyncWaitOperator.processLatencyMarker()`, thrown by `AsyncCollectorBuffer.processMark()`.
3. `AsyncWaitOperator.close()`, thrown by `AsyncWaitOperator.waitEmpty()`.
The first two conditions are for the case when not all inputs have arrived yet. The last one is for closing the operator while processing finite streaming job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86507678
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
--- End diff --
I think adding `AsyncStreamElement` has the same effect adding `AsyncCollector` for each `Watermark` and `LatencyMarker`. After all, we have to create an extra object to hold it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86308340
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private final AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ this.buffer = buffer;
--- End diff --
Maybe we could check for not null via `Preconditions.checkNotNull()`. This allows to fail early when preconditions are not fulfilled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86354718
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
--- End diff --
Could we give these `add` methods a more explicit name like `addWatermark`, `addStreamRecord`, etc. I think this makes it easier to understand that these different types are handled differently.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
The `AsyncCollectorBufferTest.testAdd` fails with a `NullPointerException`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86332678
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
--- End diff --
What about the latency marker?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
The case may happen when all elements have arrived. For this case, `StreamTask` will go to close all operators, which will reach `AsyncCollectorBuffer.waitEmpty`. In this function, `Emitter Thread` will be waken up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88720498
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java ---
@@ -116,7 +116,12 @@ public OperatorChain(StreamTask<OUT, OP> containingTask) {
// add head operator to end of chain
allOps.add(headOperator);
-
+
+ // reverse the order of all operators so that head operator is at the first place.
+ // for chained operator with async wait operator, operators after wait operator have to
+ // wait for while until all data in the buffer in wait operator has done snapshot.
+ Collections.reverse(allOps);
--- End diff --
This won't work with emitting elements in the open method of `AsyncWaitOperator`, because then the downstream operators are potentially not yet opened when the first stream element arrives there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83382559
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer, boolean isDone) {
+ this(buffer);
+ this.isDone = isDone;
+ }
+
+ /**
+ * Set result
+ * @param result A list of results.
+ */
+ public void collect(List<OUT> result) {
+ this.result = result;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+ * Set error
+ * @param error A Throwable object.
+ */
+ public void collect(Throwable error) {
+ this.error = error;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+ * Get result. Throw RuntimeException while encountering an error.
+ *
+ * @return A List of result.
+ * @throws RuntimeException RuntimeException wrapping errors from user codes.
+ */
+ public List<OUT> getResult() throws RuntimeException {
--- End diff --
I would suggest to change this to an `IOException`, or even a newly created `AsyncIOException`. The reason is that `RuntimeException` is usually used for "programming errors", while checked exceptions should be used for all (recoverable) errors that can occur due to problems by other components, or interaction with external services. http://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html?cm_mc_uid=25740090954614764351596&cm_mc_sid_50200000=1476435159
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88861548
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+
+ buffer.addWatermark(new Watermark(0l));
+ buffer.addLatencyMarker(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(buffer.getQueue().size(), 2);
+
+ Iterator<Map.Entry<AsyncCollector<Integer, Integer>, StreamElement>> iterator =
+ buffer.getQueue().entrySet().iterator();
+ Watermark watermark = iterator.next().getValue().asWatermark();
+ Assert.assertEquals(watermark.getTimestamp(), 0l);
+
+ LatencyMarker latencyMarker = iterator.next().getValue().asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp()) {
+ sb.append(record.getTimestamp() + ",");
+ }
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs) {
+ sb.append(i).append(",");
+ }
+ return sb.toString();
+ }
+
+ public Collection<Long> getOutputs() {
+ return outputs;
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ private void work(final boolean throwExcept) throws Exception {
+ final int ASYNC_COLLECTOR_NUM = 7;
+
+ final int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ final int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*100;
+
+ if (i == 3) {
+ synchronized (lock) {
+ buffer.addWatermark(new Watermark(333l));
--- End diff --
I think this should be done a little bit more explicit than by encoding a watermark emission by an integer value. Why don't you provide an `Iterable` of `StreamElements` which are inserted into the `AsyncCollectorBuffer`? Then you have full control over what goes into the `AsyncCollectorBuffer`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86486055
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86353453
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private final AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer, boolean isDone) {
+ this(buffer);
+ this.isDone = isDone;
+ }
+
+ /**
+ * Set result
+ * @param result A list of results.
+ */
+ public void collect(List<OUT> result) {
--- End diff --
The `AsyncCollector` can effectively only be called once, can't it? So unlike a `Collector` whose `collect` method can be called multiple times, this collector behaves differently. If I'm not mistaken, you're not allow to call this method multiple times. Otherwise you risk to lose data. I think this should be somehow enforced and properly documented. One could for example have a `collect` method which can be called multiple times and then a `finish` method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88707393
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
--- End diff --
`lock.wait` without `synchronized` block
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88714017
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
--- End diff --
Maybe we could use `assert(Thread.holdsLock(lock))` to guard the precondition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88707416
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
--- End diff --
why `this`? I think we should try to be consistent how we access member fields. I personally prefer not using this because it's clear from the context that it has to be a member field but you can also do it differently as long as it is consistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86520537
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/api/StreamingOperatorsITCase.java ---
@@ -195,6 +202,70 @@ public Integer map(NonSerializable value) throws Exception {
env.execute();
}
+ @Test
+ public void testAsyncWaitOperator() throws Exception {
+ final int numElements = 10;
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ DataStream<Tuple2<Integer, NonSerializable>> input = env.addSource(new NonSerializableTupleSource(numElements)).setParallelism(1);
+
+ AsyncFunction<Tuple2<Integer, NonSerializable>, Integer> function = new RichAsyncFunction<Tuple2<Integer, NonSerializable>, Integer>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(numElements);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ }
+
+ @Override
+ public void asyncInvoke(final Tuple2<Integer, NonSerializable> input,
+ final AsyncCollector<Tuple2<Integer, NonSerializable>, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * 1000);
+ try {
+ Thread.sleep(sleep);
+ List<Integer> ret = new ArrayList<>();
+ ret.add(input.f0+input.f0);
+ collector.collect(ret);
+ }
+ catch (InterruptedException e) {
+ collector.collect(new ArrayList<Integer>(0));
+ }
+ }
+ });
+ }
+ };
+
+ DataStream<Integer> orderedResult = AsyncDataStream.orderedWait(input, function, 2).setParallelism(1);
+ orderedResult.writeAsText(resultPath1, FileSystem.WriteMode.OVERWRITE).setParallelism(1);
+
+ DataStream<Integer> unorderedResult = AsyncDataStream.unorderedWait(input, function, 2).setParallelism(1);
+ unorderedResult.writeAsText(resultPath2, FileSystem.WriteMode.OVERWRITE);
--- End diff --
I think it is determined by `StreamingMultipleProgramsTestBase`, which requires tmp files from test job's output to verify the result.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86196983
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(sleepFactor*20, TimeUnit.MILLISECONDS);
--- End diff --
Any special reason why factor of 20?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88680162
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
--- End diff --
`PublicEvolving`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86335326
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
+ stream.write(input.size());
+
+ for (StreamElement element : input) {
--- End diff --
The `StreamElement` should be serialized using the `StreamElementSerializer`. Furthermore, I think it's better to wrap `stream` into a `DataOutputViewStreamWrapper` to obtain a `DataOutputView`. Then we could get rid of the `outputSerializer`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86528549
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -540,15 +540,12 @@ private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws
synchronized (lock) {
if (isRunning) {
+ checkpointState(checkpointMetaData);
- // Since both state checkpointing and downstream barrier emission occurs in this
- // lock scope, they are an atomic operation regardless of the order in which they occur.
- // Given this, we immediately emit the checkpoint barriers, so the downstream operators
- // can start their checkpoint work as soon as possible
+ // broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
- checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp());
-
- checkpointState(checkpointMetaData);
+ checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
+ );
--- End diff --
Disabling chaining does not necessarily mean that the data is sent over the network then. Flink will still try to schedule the tasks to the same machine.
Doing a benchmark is a good idea. I think we've also done it in the past and we concluded that the reentrant lock is too expensive if I remember correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86486482
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
--- End diff --
Yes, it can. We can make the executorService static
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86346855
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
--- End diff --
I see the point of using `Whitebox` here. I'm just wondering whether we could refactor the test in such a way, that it won't immediately fail when we change an internal name of `AsyncCollectorBuffer`. I think hardcoding the field names is really fragile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88700937
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
--- End diff --
`mode` could also be `null`. I think it's easier to follow the following pattern:
```
this.mode = Preconditions.checkNotNull(mode, "...");
this.output = Preconditions.checkNotNull(output, "...");
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88859015
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+
+ buffer.addWatermark(new Watermark(0l));
+ buffer.addLatencyMarker(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(buffer.getQueue().size(), 2);
+
+ Iterator<Map.Entry<AsyncCollector<Integer, Integer>, StreamElement>> iterator =
+ buffer.getQueue().entrySet().iterator();
+ Watermark watermark = iterator.next().getValue().asWatermark();
+ Assert.assertEquals(watermark.getTimestamp(), 0l);
+
+ LatencyMarker latencyMarker = iterator.next().getValue().asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp()) {
+ sb.append(record.getTimestamp() + ",");
+ }
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs) {
+ sb.append(i).append(",");
+ }
+ return sb.toString();
+ }
+
+ public Collection<Long> getOutputs() {
+ return outputs;
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ private void work(final boolean throwExcept) throws Exception {
+ final int ASYNC_COLLECTOR_NUM = 7;
+
+ final int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ final int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ ExecutorService service = Executors.newFixedThreadPool(10);
--- End diff --
Can this executor service be shared by all tests? E.g. making it static.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86198394
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
--- End diff --
Would it make sense to use an infinite source instead of a finite source? Streaming programs are in general infinite and, thus, I think it would be nice if the examples would be the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86358379
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
--- End diff --
`queue.get(i)` has a complexity of O(n) for each call. Better to do use an `Iterator` or keeping the current node to access the successor in O(1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86311659
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
--- End diff --
We could initialize the `ArrayList` with the exact size (number of elements in `queue`), then we don't have to enlarge it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88859163
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+
+ buffer.addWatermark(new Watermark(0l));
+ buffer.addLatencyMarker(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(buffer.getQueue().size(), 2);
+
+ Iterator<Map.Entry<AsyncCollector<Integer, Integer>, StreamElement>> iterator =
+ buffer.getQueue().entrySet().iterator();
+ Watermark watermark = iterator.next().getValue().asWatermark();
+ Assert.assertEquals(watermark.getTimestamp(), 0l);
+
+ LatencyMarker latencyMarker = iterator.next().getValue().asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp()) {
+ sb.append(record.getTimestamp() + ",");
+ }
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs) {
+ sb.append(i).append(",");
+ }
+ return sb.toString();
+ }
+
+ public Collection<Long> getOutputs() {
+ return outputs;
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ private void work(final boolean throwExcept) throws Exception {
+ final int ASYNC_COLLECTOR_NUM = 7;
+
+ final int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ final int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*100;
+
+ if (i == 3) {
+ synchronized (lock) {
+ buffer.addWatermark(new Watermark(333l));
+ }
+ }
+ else if (i == 6) {
+ synchronized (lock) {
+ buffer.addLatencyMarker(new LatencyMarker(111L, 0, 0));
+ }
+ }
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+
+ AsyncCollector tmp;
+ synchronized (lock) {
+ tmp = buffer.addStreamRecord(record);
+ }
+
+ final AsyncCollector collector = tmp;
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+
+ if (throwExcept) {
+ collector.collect(new Exception("wahahahaha..."));
+ }
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (InterruptedException e) {
+ }
+ }
+ });
+ }
+ }
+
+ synchronized (lock) {
+ buffer.waitEmpty();
+ }
--- End diff --
Why do you wait here for the buffer to be empty?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
The `notifyCheckpointComplete` will mitigate the problem of adding latency a little bit but not get rid of it because the tasks are only notified about a completed checkpoint if the whole checkpoint (all operators have finished their checkpoints) has succeeded. This can take a noticeable amount of time if you have a complex topology.
I really would not like to build in a hand tailored solution for the `AsyncWaitOperator` into the `StreamTask`. This is imo not a good design, because the `StreamTask` should not know anything about the operators which are executed within it.
I still think that the checkpoint lock would be a better solution because then the `Emitter` thread can continue emitting elements right after the checkpoint (local) has completed and we don't have to introduce custom control logic for the `AsyncWaitOperator`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86507065
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
--- End diff --
I changed into ArrayDeque, which is faster than LinkedList while visiting all elements.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86340921
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -540,15 +540,12 @@ private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws
synchronized (lock) {
if (isRunning) {
+ checkpointState(checkpointMetaData);
- // Since both state checkpointing and downstream barrier emission occurs in this
- // lock scope, they are an atomic operation regardless of the order in which they occur.
- // Given this, we immediately emit the checkpoint barriers, so the downstream operators
- // can start their checkpoint work as soon as possible
+ // broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
- checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp());
-
- checkpointState(checkpointMetaData);
+ checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
+ );
--- End diff --
Changing the order of broadcasting the checkpoint barrier to downstream operators and performing the actual checkpoint is very problematic. First drawing the checkpoint and then sending the checkpoint barrier downstream will add the latency of creating the checkpoint to the barrier for each operator. This will considerably influence the overall checkpointing time, because the individual checkpoints are effectively executed sequentially. Thus, I fear that this change is not possible.
Is the problem that the `AsyncWaitOperator#Emitter` can emit results while we're drawing the checkpoints? I think the proper way to solve this problem is to let the output operation happen under the checkpoint lock. That way, you are sure that no element will be outputted while the checkpointing is active.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86310348
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
--- End diff --
"until checkpointing is done"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86511575
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
--- End diff --
This is true. But conceptually it does not make sense to have a collector for a watermark or latency marker. I think it's a bit of a hack to create an already completed `AsyncCollector` for these elements.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86197630
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(sleepFactor*20, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, String> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * sleepFactor);
--- End diff --
Shall we create one `Random` instance and reuse it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83382895
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private int bufferSize;
+
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private Lock lock;
+ private Condition notFull;
+ private Condition taskDone;
+ private Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private Emitter emitter;
+ private Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, watermark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ if (emitter != null) {
--- End diff --
`emitter` and `emitThread` can probably never be `null`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83380141
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
--- End diff --
Can you put the `Thread.sleep(...)` outside the lock? Otherwise, it will be hard for the checkpoint to ever acquire the lock.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86514152
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
--- End diff --
This is a bug. Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86516426
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
--- End diff --
But then you could prohibit to add new `AsyncCollectors` to the `queue` if there was an `error`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88718312
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
+
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setOutputMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ @VisibleForTesting
+ public void setEmitFlag(boolean emitFlag) {
+ this.emitFlag = emitFlag;
+ }
+
+ @VisibleForTesting
+ public void setInStreamElementSerializerForTest(StreamElementSerializer<IN> serializer) {
+ this.inStreamElementSerializerForTest = serializer;
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ if (this.inStreamElementSerializerForTest != null) {
+ this.inStreamElementSerializer = this.inStreamElementSerializerForTest;
+ }
+ else {
+ this.inStreamElementSerializer = new StreamElementSerializer(this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader()));
--- End diff --
Raw usage of `StreamElementSerializer`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89741541
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
+
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setOutputMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ @VisibleForTesting
+ public void setEmitFlag(boolean emitFlag) {
+ this.emitFlag = emitFlag;
+ }
+
+ @VisibleForTesting
+ public void setInStreamElementSerializerForTest(StreamElementSerializer<IN> serializer) {
+ this.inStreamElementSerializerForTest = serializer;
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ if (this.inStreamElementSerializerForTest != null) {
+ this.inStreamElementSerializer = this.inStreamElementSerializerForTest;
+ }
+ else {
+ this.inStreamElementSerializer = new StreamElementSerializer(this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader()));
+ }
+
+ this.collector = new TimestampedCollector<>(output);
+
+ this.checkpointLock = containingTask.getCheckpointLock();
+
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, output, collector, this.checkpointLock, this);
+
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.recoveredStreamElements != null) {
+ for (StreamElement element : this.recoveredStreamElements.get()) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ }
+ else if (element.isWatermark()) {
+ processWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ processLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown record type: "+element.getClass());
+ }
+ }
+ this.recoveredStreamElements = null;
+ }
+
+
+ if (emitFlag) {
+ buffer.startEmitterThread();
+ }
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.addStreamRecord(element);
+
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.addWatermark(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.addLatencyMarker(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ ListState<StreamElement> partitionableState =
+ getOperatorStateBackend().getOperatorState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer));
+ partitionableState.clear();
+
+ Collection<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ for (StreamElement element : elements) {
+ partitionableState.add(element);
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ recoveredStreamElements =
+ context.getManagedOperatorStateStore().getOperatorState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer));
+
+ }
+
+ @Override
+ public void close() throws Exception {
+ // for test only, we do not have to wait until buffer is empty.
+ if (!emitFlag) {
+ return;
+ }
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
--- End diff --
Emm, I think you are right. We'd better stop the emitter thread while closing the operator, so that the order to dispose operators can still be from tail to head - no modification to the code in `StreamTask.tryDisposeAllOperators()`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88701332
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
--- End diff --
don't you have to be in a synchronized block to call `wait`? Even if you implicitly are, you should make it more explicit via a `synchronized` block.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86515304
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
+ Thread.sleep(1000);
--- End diff --
I think that hardcoded sleeps are in general not a good idea. How does `processError` notifies the main thread? Doesn't the main thread detects the error as long as the `error` field is set?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86482845
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
--- End diff --
It is optional here ;D. I will remove it later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88872044
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
--- End diff --
I think this method should make properly sure that the emitter thread has terminated. E.g. joining with a defined timeout and then doing proper reporting in case of an error.
Furthermore, I think the emitter thread should be a daemon, otherwise the JVM won't exit if the stopping should fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89311161
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
--- End diff --
No I meant the way you access member fields of a class. Sometimes you prefix them with `this` and sometimes not. I think we should be consistent in any way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86201565
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setMode(mode);
+
+ OneInputTransformation<IN, OUT> resultTransform = new OneInputTransformation<>(
+ in.getTransformation(),
+ "async wait operator",
+ operator,
+ outTypeInfo,
+ in.getExecutionEnvironment().getParallelism());
+
+ SingleOutputStreamOperator<OUT> returnStream =
+ new SingleOutputStreamOperator<>(in.getExecutionEnvironment(), resultTransform);
+
+ returnStream.getExecutionEnvironment().addOperator(resultTransform);
+
+ return returnStream;
+ }
+
+ /**
+ * Add an AsyncWaitOperator. The order of output stream records may be reordered.
+ *
+ * @param in Input {@link DataStream}
+ * @param func AsyncFunction
+ * @bufSize The max number of async i/o operation that can be triggered
+ * @return A new {@link SingleOutputStreamOperator}.
+ */
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize) {
--- End diff --
Maybe we could spell out the parameter names, because they will be shown in the IDE when getting contextual help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86344377
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
--- End diff --
Why the `sleep` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83573419
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
+ }
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, String> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * sleepFactor);
+ try {
+ Thread.sleep(sleep);
+ List<String> ret = new ArrayList<>();
+ ret.add("key-"+(input%10));
+ if (new Random().nextFloat() > failRatio) {
+ collector.collect(new Exception("wahahahaha..."));
+ }
+ else {
+ collector.collect(ret);
+ }
+ }
+ catch (InterruptedException e) {
+ collector.collect(new ArrayList<String>(0));
+ }
+ }
+ });
+ }
+
+ private void readObject(java.io.ObjectInputStream in)
--- End diff --
OK, I will move it into `open()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86348018
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ });
+ }
+ }
+ }
+ catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp())
+ sb.append(record.getTimestamp()+",");
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs)
+ sb.append(i).append(",");
+ return sb.toString();
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer.startEmitterThread();
+ Watermark watermark = new Watermark(0L);
+ buffer.add(watermark);
+
+ OrderedPutThread orderedPutThread = new OrderedPutThread(buffer);
+ Thread.sleep(2000);
+
+ orderedPutThread.start();
+
+ orderedPutThread.join();
--- End diff --
Why do we need another thread to do the ordered put if the main thread waits for this other thread to complete?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86195976
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
--- End diff --
`Checkpointed` has been deprecated. Could we use the `CheckpointedFunction` instead? Then we're up to date :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88707089
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
--- End diff --
I think watermarks and latency marker should not get a `AsyncCollector` because there is nothing to collect for them. Creating a completed `AsyncCollector` seems to be a hack.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88865433
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperatorTest.java ---
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.testutils.OneShotLatch;
+import org.apache.flink.runtime.checkpoint.CheckpointMetaData;
+import org.apache.flink.runtime.checkpoint.SubtaskState;
+import org.apache.flink.runtime.operators.testutils.MockInputSplitProvider;
+import org.apache.flink.runtime.state.TaskStateHandles;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTask;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTaskTestHarness;
+import org.apache.flink.streaming.runtime.tasks.StreamMockEnvironment;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.TestHarnessUtil;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Tests for {@link AsyncWaitOperator}. These test that:
+ *
+ * <ul>
+ * <li>Process StreamRecords and Watermarks in ORDERED mode</li>
+ * <li>Process StreamRecords and Watermarks in UNORDERED mode</li>
+ * <li>Snapshot state and restore state</li>
+ * </ul>
+ */
+public class AsyncWaitOperatorTest {
+
+ public static class MyAsyncFunction extends RichAsyncFunction<Integer, Integer> {
+ transient final int SLEEP_FACTOR = 100;
+ transient final int THREAD_POOL_SIZE = 10;
+ transient ExecutorService executorService;
--- End diff --
Can't we share the `ExecutorService`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88712331
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
--- End diff --
Error message would be quite helpful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86319324
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
--- End diff --
Same here with the `IdentityHashMap`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88867574
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperatorTest.java ---
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.testutils.OneShotLatch;
+import org.apache.flink.runtime.checkpoint.CheckpointMetaData;
+import org.apache.flink.runtime.checkpoint.SubtaskState;
+import org.apache.flink.runtime.operators.testutils.MockInputSplitProvider;
+import org.apache.flink.runtime.state.TaskStateHandles;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTask;
+import org.apache.flink.streaming.runtime.tasks.OneInputStreamTaskTestHarness;
+import org.apache.flink.streaming.runtime.tasks.StreamMockEnvironment;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.TestHarnessUtil;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Tests for {@link AsyncWaitOperator}. These test that:
+ *
+ * <ul>
+ * <li>Process StreamRecords and Watermarks in ORDERED mode</li>
+ * <li>Process StreamRecords and Watermarks in UNORDERED mode</li>
+ * <li>Snapshot state and restore state</li>
+ * </ul>
+ */
+public class AsyncWaitOperatorTest {
+
+ public static class MyAsyncFunction extends RichAsyncFunction<Integer, Integer> {
+ transient final int SLEEP_FACTOR = 100;
+ transient final int THREAD_POOL_SIZE = 10;
+ transient ExecutorService executorService;
+ transient Random random;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
+ random = new Random();
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(SLEEP_FACTOR*THREAD_POOL_SIZE, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void asyncInvoke(final Integer input, final AsyncCollector<Integer, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (random.nextFloat() * SLEEP_FACTOR);
+ try {
+ Thread.sleep(sleep);
+ List<Integer> ret = new ArrayList<>();
+ ret.add(input*2);
+ collector.collect(ret);
+ }
+ catch (InterruptedException e) {
+ }
+ }
+ });
+ }
+ }
+
+ @Test
+ public void testWaterMarkOrdered() throws Exception {
+ testWithWatermark(AsyncDataStream.OutputMode.ORDERED);
+ }
+
+ @Test
+ public void testWaterMarkUnordered() throws Exception {
+ testWithWatermark(AsyncDataStream.OutputMode.UNORDERED);
+ }
+
+ private void testWithWatermark(AsyncDataStream.OutputMode mode) throws Exception {
+ final AsyncWaitOperator<Integer, Integer> operator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ operator.setOutputMode(mode);
+ operator.setBufferSize(6);
+ operator.setInStreamElementSerializerForTest(new StreamElementSerializer<>(IntSerializer.INSTANCE));
+
+ final OneInputStreamOperatorTestHarness<Integer, Integer> testHarness =
+ new OneInputStreamOperatorTestHarness<>(operator);
+
+ final long initialTime = 0L;
+ final ConcurrentLinkedQueue<Object> expectedOutput = new ConcurrentLinkedQueue<>();
+
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(1, initialTime + 1));
+ testHarness.processElement(new StreamRecord<>(2, initialTime + 2));
+ testHarness.processWatermark(new Watermark(initialTime + 2));
+ testHarness.processElement(new StreamRecord<>(3, initialTime + 3));
+
+ // wait until all async collectors in the buffer have been emitted out.
+ synchronized (testHarness.getCheckpointLock()) {
+ testHarness.close();
+ }
+
+ expectedOutput.add(new StreamRecord<>(2, initialTime + 1));
+ expectedOutput.add(new StreamRecord<>(4, initialTime + 2));
+ expectedOutput.add(new Watermark(initialTime + 2));
+ expectedOutput.add(new StreamRecord<>(6, initialTime + 3));
+
+ if (AsyncDataStream.OutputMode.ORDERED == mode) {
+ TestHarnessUtil.assertOutputEquals("Output with watermark was not correct.", expectedOutput, testHarness.getOutput());
+ }
+ else {
+ Assert.assertTrue(expectedOutput.size() == 4);
+ Assert.assertTrue(expectedOutput.toArray(new StreamElement[4])[2].asWatermark().getTimestamp() == (initialTime+2));
+ }
+ }
+
+ @Test
+ public void testOrdered() throws Exception {
+ testRun(AsyncDataStream.OutputMode.ORDERED);
+ }
+
+ @Test
+ public void testUnordered() throws Exception {
+ testRun(AsyncDataStream.OutputMode.UNORDERED);
+ }
+
+ private void testRun(AsyncDataStream.OutputMode mode) throws Exception {
+ final OneInputStreamTask<Integer, Integer> task = new OneInputStreamTask<>();
+ final OneInputStreamTaskTestHarness<Integer, Integer> testHarness =
+ new OneInputStreamTaskTestHarness<>(task, 1, 1, BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO);
+
+ final AsyncWaitOperator<Integer, Integer> operator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ operator.setOutputMode(mode);
+ operator.setBufferSize(6);
+
+ final StreamConfig streamConfig = testHarness.getStreamConfig();
+ streamConfig.setStreamOperator(operator);
+
+ testHarness.invoke();
+ testHarness.waitForTaskRunning();
+
+ final long initialTime = 0L;
+ final ConcurrentLinkedQueue<Object> expectedOutput = new ConcurrentLinkedQueue<Object>();
+
+ testHarness.processElement(new StreamRecord<>(1, initialTime + 1));
+ testHarness.processElement(new StreamRecord<>(2, initialTime + 2));
+ testHarness.processElement(new StreamRecord<>(3, initialTime + 3));
+ testHarness.processElement(new StreamRecord<>(4, initialTime + 4));
+ testHarness.processElement(new StreamRecord<>(5, initialTime + 5));
+ testHarness.processElement(new StreamRecord<>(6, initialTime + 6));
+ testHarness.processElement(new StreamRecord<>(7, initialTime + 7));
+ testHarness.processElement(new StreamRecord<>(8, initialTime + 8));
+
+ expectedOutput.add(new StreamRecord<>(2, initialTime + 1));
+ expectedOutput.add(new StreamRecord<>(4, initialTime + 2));
+ expectedOutput.add(new StreamRecord<>(6, initialTime + 3));
+ expectedOutput.add(new StreamRecord<>(8, initialTime + 4));
+ expectedOutput.add(new StreamRecord<>(10, initialTime + 5));
+ expectedOutput.add(new StreamRecord<>(12, initialTime + 6));
+ expectedOutput.add(new StreamRecord<>(14, initialTime + 7));
+ expectedOutput.add(new StreamRecord<>(16, initialTime + 8));
+
+ testHarness.waitForInputProcessing();
+
+ testHarness.endInput();
+
+ testHarness.waitForTaskCompletion();
+
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ TestHarnessUtil.assertOutputEquals("ORDERED Output was not correct.", expectedOutput, testHarness.getOutput());
+ }
+ else {
+ TestHarnessUtil.assertOutputEqualsSorted(
+ "UNORDERED Output was not correct.",
+ expectedOutput,
+ testHarness.getOutput(),
+ new Comparator<Object>() {
+ @Override
+ public int compare(Object o1, Object o2) {
+ Integer val1 = ((StreamRecord<Integer>)o1).getValue();
+ Integer val2 = ((StreamRecord<Integer>)o2).getValue();
+ if (val1 > val2) {
+ return 1;
+ }
+ else if (val1 == val2) {
+ return 0;
+ }
+ else {
+ return -1;
+ }
+ }
+ });
+ }
+ }
+
+ @Test
+ public void testStateSnapshotAndRestore() throws Exception {
+ final OneInputStreamTask<Integer, Integer> task = new OneInputStreamTask<>();
+ final OneInputStreamTaskTestHarness<Integer, Integer> testHarness =
+ new OneInputStreamTaskTestHarness<>(task, 1, 1, BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO);
+
+ final AsyncWaitOperator<Integer, Integer> operator = new AsyncWaitOperator<>(new MyAsyncFunction());
+ operator.setOutputMode(AsyncDataStream.OutputMode.ORDERED);
+ operator.setBufferSize(6);
+ // disable starting emitter thread, so that all inputs will be kept in the buffer, which will be
+ // snapshotted into operator state
+ operator.setEmitFlag(false);
+
+ final StreamConfig streamConfig = testHarness.getStreamConfig();
+ streamConfig.setStreamOperator(operator);
+
+ final AcknowledgeStreamMockEnvironment env = new AcknowledgeStreamMockEnvironment(
+ testHarness.jobConfig,
+ testHarness.taskConfig,
+ testHarness.getExecutionConfig(),
+ testHarness.memorySize,
+ new MockInputSplitProvider(),
+ testHarness.bufferSize);
+
+ testHarness.invoke(env);
+ testHarness.waitForTaskRunning();
+
+ final long initialTime = 0L;
+
+ testHarness.processElement(new StreamRecord<>(1, initialTime + 1));
+ testHarness.processElement(new StreamRecord<>(2, initialTime + 2));
+ testHarness.processElement(new StreamRecord<>(3, initialTime + 3));
+ testHarness.processElement(new StreamRecord<>(4, initialTime + 4));
+
+ testHarness.waitForInputProcessing();
+
+ final long checkpointId = 1L;
+ final long checkpointTimestamp = 1L;
+
+ final CheckpointMetaData checkpointMetaData = new CheckpointMetaData(checkpointId, checkpointTimestamp);
+
+ while(!task.triggerCheckpoint(checkpointMetaData));
--- End diff --
Why do you do this here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88714427
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector. Try to emit results prior to the oldest watermark
+ * in the queue.
+ * <p>
+ * For example, assume the queue layout is:
+ * Entry(ac1, record1) -> Entry(ac2, record2) -> Entry(ac3, watermark1) -> Entry(ac4, record3).
+ * and both of ac2 and ac3 have finished. For unordered-mode, ac1 and ac2 are prior to watermark1,
+ * so ac2 will be emitted. Since ac1 is not ready yet, ac3 have to wait until ac1 is done.
+ */
+ private void unorderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean hasUnfinishedCollector = false;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ StreamElement element = entry.getValue();
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (element.isWatermark() && hasUnfinishedCollector) {
+ break;
+ }
+
+ if (collector.isDone() == false) {
+ hasUnfinishedCollector = true;
+
+ continue;
+ }
+
+ try {
+ output(collector, element);
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors prior to the oldest water mark.
+ * or
+ * The first element in the queue is Watermark or LatencyMarker.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ return true;
+ }
+
+ // get the first AsyncCollector and StreamElement in the queue.
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> firstEntry = queue.entrySet().iterator().next();
+ StreamElement element = firstEntry.getValue();
+ AsyncCollector<IN, OUT> collector = firstEntry.getKey();
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (element.isWatermark() || element.isLatencyMarker()) {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ return !collector.isDone();
+ }
+ else {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean noFinishedCollector = true;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ if (entry.getKey().isDone()) {
+ noFinishedCollector = false;
+ break;
--- End diff --
we could also write `return false`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88676202
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
--- End diff --
Then let's do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86347509
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
--- End diff --
Can we decrease the sleep timeouts, because otherwise the test takes a long time to run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86361495
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
--- End diff --
Is the `Emitter` supposed to continue working after an error has occurred? If not, then we should probably stop the thread by leaving the while loop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86706353
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ });
+ }
+ }
+ }
+ catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp())
+ sb.append(record.getTimestamp()+",");
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs)
+ sb.append(i).append(",");
+ return sb.toString();
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer.startEmitterThread();
+ Watermark watermark = new Watermark(0L);
+ buffer.add(watermark);
+
+ OrderedPutThread orderedPutThread = new OrderedPutThread(buffer);
+ Thread.sleep(2000);
+
+ orderedPutThread.start();
+
+ orderedPutThread.join();
--- End diff --
That is right, we can keep putting the elements. The problem is that when not all test inputs are put into the `AsyncCollectorBuffer`, the UT code may process and come through `AsyncWaitOperator.waitEmpty()` if we do not `join` the `PutThread` to emit all test inputs, leading to the undetermined results.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88681032
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
--- End diff --
line break missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86916022
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ });
+ }
+ }
+ }
+ catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp())
+ sb.append(record.getTimestamp()+",");
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs)
+ sb.append(i).append(",");
+ return sb.toString();
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer.startEmitterThread();
+ Watermark watermark = new Watermark(0L);
+ buffer.add(watermark);
+
+ OrderedPutThread orderedPutThread = new OrderedPutThread(buffer);
+ Thread.sleep(2000);
+
+ orderedPutThread.start();
+
+ orderedPutThread.join();
--- End diff --
It only concerns that whether all **input elements** are in the buffer, by `addStreamRecord` / `addWatermark` / `addLatencyMarker` in the `run()` method in `OrderedPutThread`. As long as they are in the buffer, which is ensured by `join` statement, `waitEmpty()` will wait there until all `Runnable`s exit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88676017
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.ListCheckpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, ListCheckpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public List<Integer> snapshotState(long checkpointId, long timestamp) throws Exception {
+ return Collections.singletonList(start);
+ }
+
+ @Override
+ public void restoreState(List<Integer> state) throws Exception {
+ for (Integer i : state)
+ this.start = i;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
--- End diff --
Can we set the `counter` to `-1` to indicate an infinite input stream?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86350872
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/api/StreamingOperatorsITCase.java ---
@@ -195,6 +202,70 @@ public Integer map(NonSerializable value) throws Exception {
env.execute();
}
+ @Test
+ public void testAsyncWaitOperator() throws Exception {
+ final int numElements = 10;
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ DataStream<Tuple2<Integer, NonSerializable>> input = env.addSource(new NonSerializableTupleSource(numElements)).setParallelism(1);
+
+ AsyncFunction<Tuple2<Integer, NonSerializable>, Integer> function = new RichAsyncFunction<Tuple2<Integer, NonSerializable>, Integer>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(numElements);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ }
+
+ @Override
+ public void asyncInvoke(final Tuple2<Integer, NonSerializable> input,
+ final AsyncCollector<Tuple2<Integer, NonSerializable>, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * 1000);
+ try {
+ Thread.sleep(sleep);
+ List<Integer> ret = new ArrayList<>();
+ ret.add(input.f0+input.f0);
+ collector.collect(ret);
+ }
+ catch (InterruptedException e) {
+ collector.collect(new ArrayList<Integer>(0));
+ }
+ }
+ });
+ }
+ };
+
+ DataStream<Integer> orderedResult = AsyncDataStream.orderedWait(input, function, 2).setParallelism(1);
+ orderedResult.writeAsText(resultPath1, FileSystem.WriteMode.OVERWRITE).setParallelism(1);
+
+ DataStream<Integer> unorderedResult = AsyncDataStream.unorderedWait(input, function, 2).setParallelism(1);
+ unorderedResult.writeAsText(resultPath2, FileSystem.WriteMode.OVERWRITE);
--- End diff --
It is better to not write things on disk because this makes the test case more reliable against disk problems. Better to use a sink which keeps the state in memory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86333056
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
+ stream.write(input.size());
+
+ for (StreamElement element : input) {
+ if (element.isRecord()) {
+ stream.write(1);
--- End diff --
Can we define an enum for these values?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
Update the PR based on the latest review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83571326
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
--- End diff --
That is right. It will block checkpoint for extra 0.1 second.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86492545
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setMode(mode);
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88680106
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/async/AsyncFunction.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.async;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+
+import java.io.Serializable;
+
+/**
+ * A function to trigger Async I/O operation.
+ * <p>
+ * For each #asyncInvoke, an async io operation can be triggered, and once it has been done,
+ * the result can be collected by calling {@link AsyncCollector#collect}. For each async
+ * operations, their contexts are buffered in the operator immediately after invoking
+ * #asyncInvoke, leading to no blocking for each stream input as long as internal buffer is not full.
+ * <p>
+ * {@link AsyncCollector} can be passed into callbacks or futures provided by async client to
+ * fetch result data. Any error can also be propagate to the operator by {@link AsyncCollector#collect(Throwable)}.
+ *
+ * <p>
+ * Typical usage for callback:
+ * <pre>{@code
+ * public class HBaseAsyncFunc implements AsyncFunction<String, String> {
+ * @Override
+ * public void asyncInvoke(String row, AsyncCollector<String> collector) throws Exception {
+ * HBaseCallback cb = new HBaseCallback(collector);
+ * Get get = new Get(Bytes.toBytes(row));
+ * hbase.asyncGet(get, cb);
+ * }
+ * }
+ * }
+ * </pre>
+ *
+ * <p>
+ * Typical usage for {@link com.google.common.util.concurrent.ListenableFuture}
+ * <pre>{@code
+ * public class HBaseAsyncFunc implements AsyncFunction<String, String> {
+ * @Override
+ * public void asyncInvoke(String row, final AsyncCollector<String> collector) throws Exception {
+ * Get get = new Get(Bytes.toBytes(row));
+ * ListenableFuture<Result> future = hbase.asyncGet(get);
+ * Futures.addCallback(future, new FutureCallback<Result>() {
+ * public void onSuccess(Result result) {
+ * List<String> ret = process(result);
+ * collector.collect(ret);
+ * }
+ * public void onFailure(Throwable thrown) {
+ * collector.collect(thrown);
+ * }
+ * });
+ * }
+ * }
+ * }
+ * </pre>
+ *
+ * @param <IN> The type of the input elements.
+ * @param <OUT> The type of the returned elements.
+ */
+public interface AsyncFunction<IN, OUT> extends Function, Serializable {
--- End diff --
`PublicEvolving`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88868024
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
+
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setOutputMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ @VisibleForTesting
+ public void setEmitFlag(boolean emitFlag) {
+ this.emitFlag = emitFlag;
+ }
+
+ @VisibleForTesting
+ public void setInStreamElementSerializerForTest(StreamElementSerializer<IN> serializer) {
+ this.inStreamElementSerializerForTest = serializer;
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ if (this.inStreamElementSerializerForTest != null) {
+ this.inStreamElementSerializer = this.inStreamElementSerializerForTest;
+ }
+ else {
+ this.inStreamElementSerializer = new StreamElementSerializer(this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader()));
+ }
+
+ this.collector = new TimestampedCollector<>(output);
+
+ this.checkpointLock = containingTask.getCheckpointLock();
+
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, output, collector, this.checkpointLock, this);
+
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.recoveredStreamElements != null) {
+ for (StreamElement element : this.recoveredStreamElements.get()) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
--- End diff --
Element emission in the open method only works if a chain is opened from the tail to head operator. We should have a test guarding this condition. Furthermore, a comment should be added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86347697
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ });
+ }
+ }
+ }
+ catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp())
+ sb.append(record.getTimestamp()+",");
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs)
+ sb.append(i).append(",");
+ return sb.toString();
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer.startEmitterThread();
+ Watermark watermark = new Watermark(0L);
+ buffer.add(watermark);
+
+ OrderedPutThread orderedPutThread = new OrderedPutThread(buffer);
+ Thread.sleep(2000);
--- End diff --
Why the `sleep`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86493182
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setMode(mode);
+
+ OneInputTransformation<IN, OUT> resultTransform = new OneInputTransformation<>(
+ in.getTransformation(),
+ "async wait operator",
+ operator,
+ outTypeInfo,
+ in.getExecutionEnvironment().getParallelism());
+
+ SingleOutputStreamOperator<OUT> returnStream =
+ new SingleOutputStreamOperator<>(in.getExecutionEnvironment(), resultTransform);
+
+ returnStream.getExecutionEnvironment().addOperator(resultTransform);
+
+ return returnStream;
+ }
+
+ /**
+ * Add an AsyncWaitOperator. The order of output stream records may be reordered.
+ *
+ * @param in Input {@link DataStream}
+ * @param func AsyncFunction
+ * @bufSize The max number of async i/o operation that can be triggered
+ * @return A new {@link SingleOutputStreamOperator}.
+ */
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize) {
+ return addOperator(in, func, bufSize, OutputMode.UNORDERED);
+ }
+
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func) {
+ return addOperator(in, func, DEFAULT_BUFFER_SIZE, OutputMode.UNORDERED);
+ }
+
+ /**
+ * Add an AsyncWaitOperator. The order of output stream records is guaranteed to be the same as input ones.
+ *
+ * @param in Input data stream
+ * @param func {@link AsyncFunction}
+ * @bufSize The max number of async i/o operation that can be triggered
+ * @return A new {@link SingleOutputStreamOperator}.
+ */
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> orderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize) {
+ return addOperator(in, func, bufSize, OutputMode.ORDERED);
+ }
+
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> orderedWait(DataStream<IN> in,
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86511685
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
--- End diff --
I want to log the error with corresponding input stream element.
For the case that returning an empty result set is allowable, then an empty List should be passed into `AsyncCollector.collect()`. I have added more info in the doc of this function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88714777
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector. Try to emit results prior to the oldest watermark
+ * in the queue.
+ * <p>
+ * For example, assume the queue layout is:
+ * Entry(ac1, record1) -> Entry(ac2, record2) -> Entry(ac3, watermark1) -> Entry(ac4, record3).
+ * and both of ac2 and ac3 have finished. For unordered-mode, ac1 and ac2 are prior to watermark1,
+ * so ac2 will be emitted. Since ac1 is not ready yet, ac3 have to wait until ac1 is done.
+ */
+ private void unorderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean hasUnfinishedCollector = false;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ StreamElement element = entry.getValue();
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (element.isWatermark() && hasUnfinishedCollector) {
+ break;
+ }
+
+ if (collector.isDone() == false) {
+ hasUnfinishedCollector = true;
+
+ continue;
+ }
+
+ try {
+ output(collector, element);
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors prior to the oldest water mark.
+ * or
+ * The first element in the queue is Watermark or LatencyMarker.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ return true;
+ }
+
+ // get the first AsyncCollector and StreamElement in the queue.
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> firstEntry = queue.entrySet().iterator().next();
+ StreamElement element = firstEntry.getValue();
+ AsyncCollector<IN, OUT> collector = firstEntry.getKey();
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (element.isWatermark() || element.isLatencyMarker()) {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ return !collector.isDone();
+ }
+ else {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean noFinishedCollector = true;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ if (entry.getKey().isDone()) {
+ noFinishedCollector = false;
+ break;
+ }
+
+ if (entry.getValue().isWatermark()) {
+ break;
+ }
+ }
+
+ return noFinishedCollector;
+ }
+ }
+
+ private void processFinishedAsyncCollector() throws IOException {
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ orderedProcess();
+ } else {
+ unorderedProcess();
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ synchronized (lock) {
+ try {
+ while (nothingToDo()) {
+ lock.wait();
+ }
+ }
+ catch(InterruptedException e) {
+ break;
--- End diff --
can we simply set `running = false` and don't use the `break` statement?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88712552
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector. Try to emit results prior to the oldest watermark
+ * in the queue.
+ * <p>
+ * For example, assume the queue layout is:
+ * Entry(ac1, record1) -> Entry(ac2, record2) -> Entry(ac3, watermark1) -> Entry(ac4, record3).
+ * and both of ac2 and ac3 have finished. For unordered-mode, ac1 and ac2 are prior to watermark1,
+ * so ac2 will be emitted. Since ac1 is not ready yet, ac3 have to wait until ac1 is done.
+ */
+ private void unorderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean hasUnfinishedCollector = false;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ StreamElement element = entry.getValue();
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (element.isWatermark() && hasUnfinishedCollector) {
+ break;
+ }
+
+ if (collector.isDone() == false) {
+ hasUnfinishedCollector = true;
+
+ continue;
+ }
--- End diff --
Can't we do
```
if (collector.isDone) {
output
} else {
hasUnfinishedCollector = true;
}
```
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88720966
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -982,7 +980,12 @@ public void executeCheckpointing() throws Exception {
startSyncPartNano = System.nanoTime();
- for (StreamOperator<?> op : allOperators) {
+ // perform the snapshots from head operator so that all parent operators can do snapshot before their
+ // children. for chained operators without any future wait operator/async wait operator, the order
+ // is not a concern. when it comes to chained operator with those operators, all the operators acting as
+ // wait operators' children should have received all their inputs while doing snapshot.
--- End diff --
I'm not sure how relevant this is anymore, because the output should happen under the checkpoint lock and if the code is here (executing the checkpoint), then it should already have acquired this lock.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89289895
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
--- End diff --
Emm, I'd like to add `assert(Thread.holdsLock(lock));` to make it clear here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88716757
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
+
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setOutputMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
--- End diff --
Can't we give these information at creation time of the operator? Then we can mark these fields also final.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86514286
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86527403
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ });
+ }
+ }
+ }
+ catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp())
+ sb.append(record.getTimestamp()+",");
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs)
+ sb.append(i).append(",");
+ return sb.toString();
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer.startEmitterThread();
+ Watermark watermark = new Watermark(0L);
+ buffer.add(watermark);
+
+ OrderedPutThread orderedPutThread = new OrderedPutThread(buffer);
+ Thread.sleep(2000);
+
+ orderedPutThread.start();
+
+ orderedPutThread.join();
--- End diff --
But we could also do the put operation in the main thread running the test, couldn't we?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86201732
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setMode(mode);
+
+ OneInputTransformation<IN, OUT> resultTransform = new OneInputTransformation<>(
+ in.getTransformation(),
+ "async wait operator",
+ operator,
+ outTypeInfo,
+ in.getExecutionEnvironment().getParallelism());
+
+ SingleOutputStreamOperator<OUT> returnStream =
+ new SingleOutputStreamOperator<>(in.getExecutionEnvironment(), resultTransform);
+
+ returnStream.getExecutionEnvironment().addOperator(resultTransform);
+
+ return returnStream;
+ }
+
+ /**
+ * Add an AsyncWaitOperator. The order of output stream records may be reordered.
+ *
+ * @param in Input {@link DataStream}
+ * @param func AsyncFunction
+ * @bufSize The max number of async i/o operation that can be triggered
+ * @return A new {@link SingleOutputStreamOperator}.
+ */
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize) {
+ return addOperator(in, func, bufSize, OutputMode.UNORDERED);
+ }
+
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> unorderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func) {
+ return addOperator(in, func, DEFAULT_BUFFER_SIZE, OutputMode.UNORDERED);
+ }
+
+ /**
+ * Add an AsyncWaitOperator. The order of output stream records is guaranteed to be the same as input ones.
+ *
+ * @param in Input data stream
+ * @param func {@link AsyncFunction}
+ * @bufSize The max number of async i/o operation that can be triggered
+ * @return A new {@link SingleOutputStreamOperator}.
+ */
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> orderedWait(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize) {
+ return addOperator(in, func, bufSize, OutputMode.ORDERED);
+ }
+
+ public static <IN, OUT> SingleOutputStreamOperator<OUT> orderedWait(DataStream<IN> in,
--- End diff --
JavaDocs missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by mproch <gi...@git.apache.org>.
Github user mproch commented on the issue:
https://github.com/apache/flink/pull/2629
@bjlovegithub @tillrohrmann thanks for quick reply :+1: I really appreciate your work!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
For exactly once processing guarantees, when `AsyncCollectorBuffer.getStreamElementsInBuffer` has finished, `isCheckpointing` will be set to true, while will idle `Emitter Thread` while checking `nonthingToDo()`. So, once snapshotting in `StreamTask` is triggered, all `EmitterThread` have stopped there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88711691
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
--- End diff --
why calling `lock.notifyAll()` here? You still keep the lock for the rest of the iterations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88679834
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/async/RichAsyncFunction.java ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.async;
+
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.functions.RichFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+
+/**
+ * Rich variant of the {@link AsyncFunction}. As a {@link RichFunction}, it gives access to the
+ * {@link org.apache.flink.api.common.functions.RuntimeContext} and provides setup and teardown methods:
+ * {@link RichFunction#open(org.apache.flink.configuration.Configuration)} and
+ * {@link RichFunction#close()}.
+ *
+ * @param <IN> The type of the input elements.
+ * @param <OUT> The type of the returned elements.
+ */
+public abstract class RichAsyncFunction<IN, OUT> extends AbstractRichFunction
--- End diff --
I'm not sure whether we should offer a rich variant of `AsyncFunction`, because any access to state from a future callback will not be scoped to the correct key. I'm quite sure that people will run into this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88857479
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
--- End diff --
Then you could also change the history size of the latency gauge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86196750
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
--- End diff --
integer pairs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
Good point ;D
Emm, I think we have to override the `StreamOpeartor.notifyCheckpointComplete()` for `AsyncWaitOpeartor` so that once the `TaskManager` notifies the `Task` that checkpoint has completed, the `EmitterThread` can start working as soon as possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86517940
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -540,15 +540,12 @@ private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws
synchronized (lock) {
if (isRunning) {
+ checkpointState(checkpointMetaData);
- // Since both state checkpointing and downstream barrier emission occurs in this
- // lock scope, they are an atomic operation regardless of the order in which they occur.
- // Given this, we immediately emit the checkpoint barriers, so the downstream operators
- // can start their checkpoint work as soon as possible
+ // broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
- checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp());
-
- checkpointState(checkpointMetaData);
+ checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
+ );
--- End diff --
Yes. We have discussed this problem, and the solution will make sure that we will not change the order of broadcasting barriers and then snapshotting.
The inelegant way is to stop all `Emitter` Thread first before broadcasting barrier. But it is a little tricky.
Another way is to change checkpoint lock into ReentrantReadWriteLock. For main thread and emitter thread, they have to acquire read lock. But for checkpoint thread or checkpoint procedure, write lock should be taken first, so that all emitter threads also stop working. In this way, main thread and all emitter thread will not block each other.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86508413
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86486644
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ executorService.awaitTermination(sleepFactor*20, TimeUnit.MILLISECONDS);
--- End diff --
The max number of `AsyncCollector` in the buffer is 20, by default, so the max number of threads in executor service is 20.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86349629
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/api/StreamingOperatorsITCase.java ---
@@ -195,6 +202,70 @@ public Integer map(NonSerializable value) throws Exception {
env.execute();
}
+ @Test
+ public void testAsyncWaitOperator() throws Exception {
+ final int numElements = 10;
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ DataStream<Tuple2<Integer, NonSerializable>> input = env.addSource(new NonSerializableTupleSource(numElements)).setParallelism(1);
+
+ AsyncFunction<Tuple2<Integer, NonSerializable>, Integer> function = new RichAsyncFunction<Tuple2<Integer, NonSerializable>, Integer>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(numElements);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ executorService.shutdown();
+ }
+
+ @Override
+ public void asyncInvoke(final Tuple2<Integer, NonSerializable> input,
+ final AsyncCollector<Tuple2<Integer, NonSerializable>, Integer> collector) throws Exception {
+ this.executorService.submit(new Runnable() {
+ @Override
+ public void run() {
+ // wait for while to simulate async operation here
+ int sleep = (int) (new Random().nextFloat() * 1000);
--- End diff --
Decrease the sleep timeout will help to speedup the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86333498
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
+ stream.write(input.size());
+
+ for (StreamElement element : input) {
+ if (element.isRecord()) {
+ stream.write(1);
+
+ StreamRecord<IN> record = element.asRecord();
+
+ outputSerializer.clear();
+ inTypeSerializer.serialize(record.getValue(), outputSerializer);
+ outputSerializer.writeLong(record.getTimestamp());
+ outputSerializer.writeBoolean(record.hasTimestamp());
--- End diff --
We don't have to write a timestamp if there is no timestamp in the record.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86486085
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
--- End diff --
Emm, I will change it later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86507941
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
--- End diff --
But the type of `finishedCollectors` could still be `Queue` right? It's always good to use the most general type possible. This does not create additional constraints when changing the code later on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88680325
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private final AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ Preconditions.checkNotNull(buffer, "Reference to AsyncCollectorBuffer should not be null");
+
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer, boolean isDone) {
+ this(buffer);
+ this.isDone = isDone;
--- End diff --
An `AsyncCollector` should never be done when being created, shouldn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2629
Great to hear @bjlovegithub :-) Ping me once you've pushed some new commits.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
Maybe this is a solution. We will not think about changing the type of `checkpointLock`, which is an `Object` ans quite efficient, and not change the order of `broadcastBarriers` and `operator.snapshotState()`. By placing **pause** `EmitterThread` codes in `StreamTask.performCheckpoint()`, like this:
` private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws Exception {
LOG.debug("Starting checkpoint {} on task {}", checkpointMetaData.getCheckpointId(), getName());
synchronized (lock) {
if (isRunning) {
// stop working threads first.
for (StreamOperator operator : operatorChain.getAllOperators()) {
if (operator instanceof AsyncWaitOperator) {
operator.pauseEmitterThread();
}
}
// broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
);
checkpointState(checkpointMetaData);
return true;
} else {
return false;
}
}
}`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86509999
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
--- End diff --
fixed by traversing the whole `LinkedHashMap`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86200906
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize, OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setMode(mode);
--- End diff --
Maybe we could rename this method to `setOutputMode`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86759570
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
--- End diff --
But you could check in `waitEmpty` whether there was an error before entering the while loop which waits for the empty signal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86507794
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
--- End diff --
Then `AsyncCollector` should be marked `final`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86513230
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
--- End diff --
It has to signal main thread which may wait there while trying to close async wait operator. When task tries to close all operators, it has to wait for while until all `AsyncCollector` have finished. Assume that some error happens first, `processError()` is called and Emitter thread exits, then main thread finds the buffer is not empty, it will wait there permanently.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86349416
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ });
+ }
+ }
+ }
+ catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp())
+ sb.append(record.getTimestamp()+",");
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs)
+ sb.append(i).append(",");
+ return sb.toString();
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer.startEmitterThread();
+ Watermark watermark = new Watermark(0L);
+ buffer.add(watermark);
+
+ OrderedPutThread orderedPutThread = new OrderedPutThread(buffer);
+ Thread.sleep(2000);
+
+ orderedPutThread.start();
+
+ orderedPutThread.join();
+
+ buffer.waitEmpty();
+
+ buffer.stopEmitterThread();
+
+ Assert.assertEquals(((FakedOutput)output).getResult(), "0,0,1,2,333,4,5,111,");
+ Assert.assertEquals(((FakedOutput)output).getTimestamp(), "0,1,4,16,25,");
+ }
+
+ private String sort(String s) {
+ String[] arr = s.split(",");
+ int[] ints = new int[arr.length];
+ for (int i = 0; i < arr.length; ++i)
+ ints[i] = Integer.valueOf(arr[i]);
+ Arrays.sort(ints);
+ return Arrays.toString(ints);
+ }
+
+ @Test
+ public void testUnorderedBuffer() throws Exception {
+ buffer.startEmitterThread();
+ Watermark watermark = new Watermark(0L);
+ buffer.add(watermark);
+
+ OrderedPutThread orderedPutThread = new OrderedPutThread(buffer, false, false);
+ Thread.sleep(2000);
+
+ orderedPutThread.start();
+
+ orderedPutThread.join();
+
+ buffer.waitEmpty();
+
+ buffer.stopEmitterThread();
+
+ Assert.assertEquals(sort(((FakedOutput)output).getResult()), "[0, 0, 1, 2, 4, 5, 111, 333]");
+ Assert.assertEquals(sort(((FakedOutput)output).getTimestamp()), "[0, 1, 4, 16, 25]");
+ }
+
+ @Test(expected = IOException.class)
+ public void testBufferWithException() throws Exception {
+ buffer.startEmitterThread();
+
+ OrderedPutThread putThread = new OrderedPutThread(buffer, true);
+ Thread.sleep(2000);
+
+ putThread.start();
+
+ putThread.join();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
--- End diff --
The emitter thread won't be stopped because an IOException is thrown before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88715633
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ private final static String STATE_NAME = "_async_wait_operator_state_";
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient StreamElementSerializer<IN> inStreamElementSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient ListState<StreamElement> recoveredStreamElements;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ /**
+ * Checkpoint lock from {@link StreamTask#lock}
+ */
+ private transient Object checkpointLock;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ /**
+ * For test only. Normally this flag is true, indicating that the Emitter Thread
+ * in the buffer will work.
+ */
+ private boolean emitFlag = true;
+
+ /**
+ * Test serializer used in unit test
+ */
+ private StreamElementSerializer<IN> inStreamElementSerializerForTest;
--- End diff --
Can't this be solved differently? It feels wrong to add testing code to non testing classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86196580
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
--- End diff --
Why do you disable operator chaining in the at least once case and not exactly once?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88707924
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
--- End diff --
white spaces between `+`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86196908
--- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example to illustrates how to use {@link org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+ * A checkpointed source.
+ */
+ private static class SimpleSource implements SourceFunction<Integer>, Checkpointed<Integer> {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext<Integer> ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ }
+ Thread.sleep(10);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1 to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ env.setStateBackend(new FsStateBackend(statePath));
+ if (cpMode.equals("exactly_once")) {
+ env.enableCheckpointing(1000, CheckpointingMode.EXACTLY_ONCE);
+ }
+ else {
+ env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
+ env.disableOperatorChaining();
+ }
+
+ // enable watermark or not
+ if (timeType.equals("EventTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+ }
+ else if (timeType.equals("IngestionTime")) {
+ env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+ }
+
+ // create input stream of integer pairs
+ DataStream<Integer> inputStream = env.addSource(new SimpleSource(maxCount));
+
+ // create async function, which will *wait* for a while to simulate the process of async i/o
+ AsyncFunction<Integer, String> function = new RichAsyncFunction<Integer, String>() {
+ transient ExecutorService executorService;
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ executorService = Executors.newFixedThreadPool(30);
--- End diff --
Could this be shared across a TM?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86321017
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
+ try {
+ lock.lock();
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ finishedCollectors.add(collector);
+ }
+
+ taskDone.signal();
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or Exceptions from AsyncCollector.
+ */
+ void waitEmpty() throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() != 0)
+ isEmpty.await();
+
+ if (error != null) {
+ throw error;
+ }
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to isCheckpointing flag
+ * until doing checkpoint has done.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ try {
+ lock.lock();
+
+ // stop emitter thread
+ isCheckpointing = true;
+
+ List<StreamElement> ret = new ArrayList<>();
+ for (int i = 0; i < queue.size(); ++i) {
+ AsyncCollector<IN, OUT> collector = queue.get(i);
+ ret.add(collectorToStreamElement.get(collector));
+ }
+
+ return ret;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ public void setOutput(TimestampedCollector<OUT> collector, Output<StreamRecord<OUT>> output) {
+ this.timestampedCollector = collector;
+ this.output = output;
+ }
+
+ public void notifyCheckpointDone() {
+ this.isCheckpointing = false;
+ this.taskDone.signalAll();
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ StreamElement element = collectorToStreamElement.get(collector);
+ if (element == null) {
+ throw new Exception("No input stream record or watermark for current AsyncCollector: "+collector);
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new Exception("Unknown input record: "+element);
+ }
+ }
+
+ private void clearInfoInMaps(AsyncCollector collector) {
+ collectorToStreamElement.remove(collector);
+ collectorToQueue.remove(collector);
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() {
+ while (queue.size() > 0) {
+ try {
+ AsyncCollector collector = queue.get(0);
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector);
+
+ queue.remove(0);
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector.
+ */
+ private void unorderedProcess() {
+ AsyncCollector collector = finishedCollectors.pollFirst();
+ while (collector != null) {
+ try {
+ output(collector);
+
+ queue.remove(collectorToQueue.get(collector));
+ clearInfoInMaps(collector);
+
+ notFull.signal();
+
+ collector = finishedCollectors.pollFirst();
+ }
+ catch (Exception e) {
+ error = e;
+ break;
+ }
+ }
+ }
+
+ /**
+ * If some bad things happened(like exceptions from async i/o), the operator tries to fail
+ * itself at:
+ * {@link AsyncWaitOperator#processElement}, triggered by calling {@link AsyncCollectorBuffer#add}.
+ * {@link AsyncWaitOperator#snapshotState}
+ * {@link AsyncWaitOperator#close} while calling {@link AsyncCollectorBuffer#waitEmpty}
+ *
+ * It is necessary for Emitter Thread to notify methods blocking on notFull/isEmpty.
+ */
+ private void processError() {
+ queue.clear();
+ finishedCollectors.clear();
+ collectorToQueue.clear();
+ collectorToStreamElement.clear();
+
+ notFull.signalAll();
+ isEmpty.signalAll();
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors.
+ * or
+ * The first element in the queue is Watermark.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ isEmpty.signalAll();
+ return true;
+ }
+
+ // while doing checkpoints, emitter thread should not try to output records.
+ if (isCheckpointing) {
+ return true;
+ }
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (collectorToStreamElement.get(queue.get(0)).isWatermark() ||
+ collectorToStreamElement.get(queue.get(0)).isLatencyMarker())
+ {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.UNORDERED) {
+ // no finished async collector...
+ if (finishedCollectors.size() == 0) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ else {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ AsyncCollector collector = queue.get(0);
+ if (collector.isDone() == false) {
+ return true;
+ }
+ else {
+ return false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ while (running) {
+ try {
+ lock.lock();
+
+ if (error != null) {
+ // stop processing finished async collectors, and try to wake up blocked main
+ // thread or checkpoint thread repeatedly.
+ processError();
+ Thread.sleep(1000);
--- End diff --
Why do we sleep here? How did you come up with the value of 1s?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88715262
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
--- End diff --
Raw type is used here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86333378
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
+ stream.write(input.size());
+
+ for (StreamElement element : input) {
+ if (element.isRecord()) {
+ stream.write(1);
+
+ StreamRecord<IN> record = element.asRecord();
+
+ outputSerializer.clear();
+ inTypeSerializer.serialize(record.getValue(), outputSerializer);
+ outputSerializer.writeLong(record.getTimestamp());
+ outputSerializer.writeBoolean(record.hasTimestamp());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else if (element.isWatermark()) {
+ stream.write(0);
+
+ Watermark watermark = element.asWatermark();
+
+ outputSerializer.clear();
+ outputSerializer.writeLong(watermark.getTimestamp());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else if (element.isLatencyMarker()) {
+ stream.write(2);
+
+ LatencyMarker marker = element.asLatencyMarker();
+
+ outputSerializer.clear();
+ outputSerializer.writeLong(marker.getMarkedTime());
+ outputSerializer.writeInt(marker.getVertexID());
+ outputSerializer.writeInt(marker.getSubtaskIndex());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else {
+ throw new IOException("Unknown element type: "+element);
+ }
+ }
+ }
+
+ private List<StreamElement> deserializeStreamElements(FSDataInputStream stream) throws IOException {
+ DataInputViewStreamWrapper wrapper = new DataInputViewStreamWrapper(stream);
+
+ int size = wrapper.read();
+
+ List<StreamElement> ret = new ArrayList<>(size);
+ for (int i = 0; i < size; ++i) {
+ int flag = wrapper.read();
+
+ if (flag == 1) {
--- End diff --
This could be a `switch` statement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89431529
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
--- End diff --
I think about it again while optimizing `nonthingToDo` method. If we remove those `AsyncCollector`s for markers, we have to keep them in other places, probably `HashMap`, and we have to get to know whether the `StreamElement` after current `StreamRecord` is `Watermark` or `LatencyMarker`. Removing those `AsyncCollector` for markers seems not bring benefits. Instead, if we just simply keep them in the buffer and use those `AsyncCollector`, working as stubs here, to main the order of input `StreamRecord`, it will be easier for us to process them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86493343
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while processing async i/o.
+ *
+ * @param <IN> Input type
+ * @param <OUT> Output type
+ */
+@Internal
+public class AsyncCollector<IN, OUT> {
+ private List<OUT> result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private final AsyncCollectorBuffer<IN, OUT> buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer<IN, OUT> buffer) {
+ this.buffer = buffer;
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88678544
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/AsyncDataStream.java ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncWaitOperator;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+/**
+ * A helper class to apply {@link AsyncFunction} to a data stream.
+ * <p>
+ * <pre>{@code
+ * DataStream<String> input = ...
+ * AsyncFunction<String, Tuple<String, String>> asyncFunc = ...
+ *
+ * AsyncDataStream.orderedWait(input, asyncFunc, 100);
+ * }
+ * </pre>
+ */
+public class AsyncDataStream {
+ public enum OutputMode { ORDERED, UNORDERED }
+
+ private static final int DEFAULT_BUFFER_SIZE = 100;
+
+ private static <IN, OUT> SingleOutputStreamOperator<OUT> addOperator(
+ DataStream<IN> in,
+ AsyncFunction<IN, OUT> func,
+ int bufSize,
+ OutputMode mode) {
+ TypeInformation<OUT> outTypeInfo =
+ TypeExtractor.getUnaryOperatorReturnType((Function) func, AsyncFunction.class, false,
+ true, in.getType(), Utils.getCallLocationName(), true);
+
+ // create transform
+ AsyncWaitOperator<IN, OUT> operator = new AsyncWaitOperator<>(in.getExecutionEnvironment().clean(func));
+ operator.setBufferSize(bufSize);
+ operator.setOutputMode(mode);
+
+ OneInputTransformation<IN, OUT> resultTransform = new OneInputTransformation<>(
+ in.getTransformation(),
+ "async wait operator",
+ operator,
+ outTypeInfo,
+ in.getExecutionEnvironment().getParallelism());
+
+ SingleOutputStreamOperator<OUT> returnStream =
+ new SingleOutputStreamOperator<>(in.getExecutionEnvironment(), resultTransform);
+
+ returnStream.getExecutionEnvironment().addOperator(resultTransform);
+
+ return returnStream;
--- End diff --
Can't we replace everything from `resultTransform` via
```
return in.transform(
"async wait operator",
outTypeInfo,
operator);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r89289557
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
--- End diff --
I understand this. The reason why I mentioned this is that the function should not have to know anything about an operator. But with this package structure the function has to know about operators.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r92536062
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/async/RichAsyncFunction.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.async;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RichFunction;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.streaming.api.functions.async.collector.AsyncCollector;
+
+/**
+ * Rich variant of the {@link AsyncFunction}. As a {@link RichFunction}, it gives access to the
+ * {@link org.apache.flink.api.common.functions.RuntimeContext} and provides setup and teardown methods:
+ * {@link RichFunction#open(org.apache.flink.configuration.Configuration)} and
+ * {@link RichFunction#close()}.
+ *
+ * <p>
+ * {@link RichAsyncFunction#getRuntimeContext()} and {@link RichAsyncFunction#getRuntimeContext()} are
+ * not supported because the key may get changed while accessing states in the working thread.
+ *
+ * @param <IN> The type of the input elements.
+ * @param <OUT> The type of the returned elements.
+ */
+
+@PublicEvolving
+public abstract class RichAsyncFunction<IN, OUT> extends AbstractRichFunction
+ implements AsyncFunction<IN, OUT> {
+
+ @Override
+ public abstract void asyncInvoke(IN input, AsyncCollector<OUT> collector) throws Exception;
+
+ @Override
+ public RuntimeContext getRuntimeContext() {
--- End diff --
I agree. It will not miss anything except state access :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86360445
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
--- End diff --
Can't we use a `LinkedHashMap`/`LinkedHashSet` to subsume the `queue`, the `collectorToStreamElement` and the `collectorToQueue` maps? The only thing we have to do is to store the `StreamElement` in the `AsyncCollector` or we store `Tuple2<StreamElement, AsyncCollector>`.
Another benefit would be that we didn't have to copy the `LinkedList` code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86347536
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ TimestampedCollector<Integer> collector =new TimestampedCollector(output);
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, operator);
+ buffer.setOutput(collector, output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ Thread.sleep(1000);
+ buffer.add(new Watermark(0l));
+ buffer.add(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).size(), 2);
+ Assert.assertEquals(((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).size(), 2);
+
+ AsyncCollector collector = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(0);
+ Watermark ret = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector)).asWatermark();
+ Assert.assertEquals(ret.getTimestamp(), 0l);
+
+ AsyncCollector collector2 = (AsyncCollector)((SimpleLinkedList) Whitebox.getInternalState(buffer, "queue")).get(1);
+ LatencyMarker latencyMarker = ((StreamElement)((Map) Whitebox.getInternalState(buffer, "collectorToStreamElement")).get(collector2)).asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ SimpleLinkedList list = (SimpleLinkedList) Whitebox.getInternalState(buffer, "queue");
+ Assert.assertEquals(list.node(0), ((Map) Whitebox.getInternalState(buffer, "collectorToQueue")).get(collector));
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class OrderedPutThread extends Thread {
+ final int ASYNC_COLLECTOR_NUM = 7;
+ int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ AsyncCollectorBuffer<Integer, Integer> buffer;
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ boolean throwExcept = false;
+ boolean orderedMode = false;
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except, boolean orderedMode) {
+ this.buffer = buffer;
+ this.throwExcept = except;
+ this.orderedMode = orderedMode;
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer, boolean except) {
+ this(buffer, except, true);
+ }
+
+ public OrderedPutThread(AsyncCollectorBuffer buffer) {
+ this(buffer, false);
+ }
+
+ @Override
+ public void run() {
+ try {
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*1000;
+
+ if (i == 3)
+ buffer.add(new Watermark(333l));
+ else if (i == 6)
+ buffer.add(new LatencyMarker(111L, 0, 0));
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+ final AsyncCollector collector = buffer.add(record);
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+ if (throwExcept)
+ collector.collect(new Exception("wahahaha..."));
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ });
+ }
+ }
+ }
+ catch (Exception e) {
+ e.printStackTrace();
--- End diff --
Same here with the exceptions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
Hi @tillrohrmann . I have updated the code again, adding the Emitter Thread, using the latest OperatorStateStore, updating the test codes. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
Hi @tillrohrmann , Thanks for your review ;D I will check through each of your comments and update the PR later.
Coming to the first part of review, the first one is about `UNORDERED` mode against `Watermark`. This combination is meaningless, of course. Maybe an error can be printed out and the graph generator stops compiling the graph if `UNORDERED` mode and `Watermark` are enabled at the same time?
Both of these two modes are guaranteed by `AsyncWaitOperator`. While doing checkpoint for the chained operator and making the snapshot for the `AsyncWaitOperator`, it will first try to get all elements in the `AsyncCollectorBuffer` by calling `getStreamElementsInBuffer()`, which will try to get the lock first to block `Emitter` thread and set a flag named `isCheckpointing` to idle `Emitter` thread. So any finished `AsyncCollector` will not be transferred to the next operator. Calling the `snapshotState()` method is from the head operator to the tail operator, making sure that all states can be taken correctly since `Emitter` threads in parent operators have stopped working.
I used to consider about using checkpoint lock in `Emitter` thread, but after testing with the case chaining multiple `AsyncWaitOperator` together, all `Emitter` thread can not fully utilize the the parallelism since they have to get the same lock while collecting outputs. One way to optimize this is to put a conditional statement at `performCheckpoint()`, if there is an `AsyncWaitOpeartor` in the chained operator, then it should broadcast barriers later after `checkpointState()`, otherwise, we can use original design.
At last, I will add more test cases based on the `OneInputStreamTaskTestHarness`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86527618
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ---
@@ -540,15 +540,12 @@ private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws
synchronized (lock) {
if (isRunning) {
+ checkpointState(checkpointMetaData);
- // Since both state checkpointing and downstream barrier emission occurs in this
- // lock scope, they are an atomic operation regardless of the order in which they occur.
- // Given this, we immediately emit the checkpoint barriers, so the downstream operators
- // can start their checkpoint work as soon as possible
+ // broadcast barriers after snapshot operators' states.
operatorChain.broadcastCheckpointBarrier(
- checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp());
-
- checkpointState(checkpointMetaData);
+ checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
+ );
--- End diff --
Disabling chain is an option, but it is a little strict to force it. Users may try to avoid network overhead by chaining them together.
I think it is worth to have a try to introduce `ReentrantReadWriteLock`. Maybe I can have a test between `Object.lock()` and `ReentrantReadWriteLock`, to get the performance difference while locking and unlock for many times.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
For exactly once processing guarantees, when `AsyncCollectorBuffer.getStreamElementsInBuffer` has finished, `isCheckpointing` will be set to true, while will idle `Emitter Thread` while checking `nonthingToDo()`. So, once snapshotting in `StreamTask` is triggered, all `EmitterThread` have stopped there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86335519
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.runtime.util.DataOutputSerializer;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Internal
+public class AsyncWaitOperator<IN, OUT>
+ extends AbstractUdfStreamOperator<OUT, AsyncFunction<IN, OUT>>
+ implements OneInputStreamOperator<IN, OUT>
+{
+ private final int DEFAULT_BUFFER_SIZE = 1000;
+
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * {@link TypeSerializer} for inputs while making snapshots.
+ */
+ private transient TypeSerializer<IN> inTypeSerializer;
+ private transient DataOutputSerializer outputSerializer;
+
+ /**
+ * input stream elements from the state
+ */
+ private transient List<StreamElement> inputsFromState;
+
+ private transient TimestampedCollector<OUT> collector;
+
+ private transient AsyncCollectorBuffer<IN, OUT> buffer;
+
+ private int bufferSize = DEFAULT_BUFFER_SIZE;
+ private AsyncDataStream.OutputMode mode;
+
+ public AsyncWaitOperator(AsyncFunction<IN, OUT> asyncFunction) {
+ super(asyncFunction);
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ }
+
+ public void setBufferSize(int size) {
+ Preconditions.checkArgument(size > 0, "The number of concurrent async operation should be greater than 0.");
+ bufferSize = size;
+ }
+
+ public void setMode(AsyncDataStream.OutputMode mode) {
+ this.mode = mode;
+ }
+
+ public void init() {
+ this.buffer = new AsyncCollectorBuffer<>(bufferSize, mode, this);
+ this.collector = new TimestampedCollector<>(output);
+ this.buffer.setOutput(collector, output);
+
+ this.outputSerializer = new DataOutputSerializer(128);
+ }
+
+ @Override
+ public void setup(StreamTask<?, ?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+
+ this.inTypeSerializer = this.getOperatorConfig().getTypeSerializerIn1(getUserCodeClassloader());
+
+ init();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // process stream elements from state
+ if (this.inputsFromState != null) {
+ for (StreamElement element : this.inputsFromState) {
+ if (element.isRecord()) {
+ processElement(element.<IN>asRecord());
+ } else {
+ processWatermark(element.asWatermark());
+ }
+ }
+ this.inputsFromState = null;
+ }
+
+ buffer.startEmitterThread();
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ AsyncCollector<IN, OUT> collector = buffer.add(element);
+ userFunction.asyncInvoke(element.getValue(), collector);
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ buffer.add(mark);
+ }
+
+ @Override
+ public void processLatencyMarker(LatencyMarker latencyMarker) throws Exception {
+ buffer.add(latencyMarker);
+ }
+
+ @Override
+ public void snapshotState(FSDataOutputStream out, long checkpointId, long timestamp) throws Exception {
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+
+ serializeStreamElements(elements, out);
+ }
+
+ @Override
+ public void restoreState(FSDataInputStream in) throws Exception {
+ this.inputsFromState = deserializeStreamElements(in);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+ }
+
+ @Override
+ public void dispose() throws Exception {
+ super.dispose();
+
+ buffer.stopEmitterThread();
+ }
+
+ public void sendLatencyMarker(LatencyMarker marker) throws Exception {
+ super.processLatencyMarker(marker);
+ }
+
+ private void serializeStreamElements(List<StreamElement> input,
+ FSDataOutputStream stream) throws IOException {
+ stream.write(input.size());
+
+ for (StreamElement element : input) {
+ if (element.isRecord()) {
+ stream.write(1);
+
+ StreamRecord<IN> record = element.asRecord();
+
+ outputSerializer.clear();
+ inTypeSerializer.serialize(record.getValue(), outputSerializer);
+ outputSerializer.writeLong(record.getTimestamp());
+ outputSerializer.writeBoolean(record.hasTimestamp());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else if (element.isWatermark()) {
+ stream.write(0);
+
+ Watermark watermark = element.asWatermark();
+
+ outputSerializer.clear();
+ outputSerializer.writeLong(watermark.getTimestamp());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else if (element.isLatencyMarker()) {
+ stream.write(2);
+
+ LatencyMarker marker = element.asLatencyMarker();
+
+ outputSerializer.clear();
+ outputSerializer.writeLong(marker.getMarkedTime());
+ outputSerializer.writeInt(marker.getVertexID());
+ outputSerializer.writeInt(marker.getSubtaskIndex());
+
+ stream.write(outputSerializer.getCopyOfBuffer());
+ }
+ else {
+ throw new IOException("Unknown element type: "+element);
+ }
+ }
+ }
+
+ private List<StreamElement> deserializeStreamElements(FSDataInputStream stream) throws IOException {
+ DataInputViewStreamWrapper wrapper = new DataInputViewStreamWrapper(stream);
+
+ int size = wrapper.read();
+
+ List<StreamElement> ret = new ArrayList<>(size);
+ for (int i = 0; i < size; ++i) {
--- End diff --
Here we could use the `StreamElementDeserializer` to deserialize the `StreamElements`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88707993
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
--- End diff --
Maybe `IOException`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88714544
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector. Try to emit results prior to the oldest watermark
+ * in the queue.
+ * <p>
+ * For example, assume the queue layout is:
+ * Entry(ac1, record1) -> Entry(ac2, record2) -> Entry(ac3, watermark1) -> Entry(ac4, record3).
+ * and both of ac2 and ac3 have finished. For unordered-mode, ac1 and ac2 are prior to watermark1,
+ * so ac2 will be emitted. Since ac1 is not ready yet, ac3 have to wait until ac1 is done.
+ */
+ private void unorderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean hasUnfinishedCollector = false;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ StreamElement element = entry.getValue();
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (element.isWatermark() && hasUnfinishedCollector) {
+ break;
+ }
+
+ if (collector.isDone() == false) {
+ hasUnfinishedCollector = true;
+
+ continue;
+ }
+
+ try {
+ output(collector, element);
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors prior to the oldest water mark.
+ * or
+ * The first element in the queue is Watermark or LatencyMarker.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
+ if (queue.size() == 0) {
+ return true;
+ }
+
+ // get the first AsyncCollector and StreamElement in the queue.
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> firstEntry = queue.entrySet().iterator().next();
+ StreamElement element = firstEntry.getValue();
+ AsyncCollector<IN, OUT> collector = firstEntry.getKey();
+
+ // check head element of the queue, it is OK to process Watermark or LatencyMarker
+ if (element.isWatermark() || element.isLatencyMarker()) {
+ return false;
+ }
+
+ if (mode == AsyncDataStream.OutputMode.ORDERED) {
+ // for ORDERED mode, make sure the first collector in the queue has been done.
+ return !collector.isDone();
+ }
+ else {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean noFinishedCollector = true;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ if (entry.getKey().isDone()) {
+ noFinishedCollector = false;
+ break;
+ }
+
+ if (entry.getValue().isWatermark()) {
+ break;
--- End diff --
and here `return true`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88861278
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBufferTest.java ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Tests for {@link AsyncCollectorBuffer}. These test that:
+ *
+ * <ul>
+ * <li>Add a new item into the buffer</li>
+ * <li>Ordered mode processing</li>
+ * <li>Unordered mode processing</li>
+ * <li>Error handling</li>
+ * </ul>
+ */
+public class AsyncCollectorBufferTest {
+ private AsyncFunction<Integer, Integer> function;
+
+ private AsyncWaitOperator<Integer, Integer> operator;
+
+ private AsyncCollectorBuffer<Integer, Integer> buffer;
+
+ private Output<StreamRecord<Integer>> output;
+
+ private TimestampedCollector<Integer> collector;
+
+ private Object lock = new Object();
+
+ @Before
+ public void setUp() throws Exception {
+ function = new AsyncFunction<Integer, Integer>() {
+ @Override
+ public void asyncInvoke(Integer input, AsyncCollector<Integer, Integer> collector) throws Exception {
+
+ }
+ };
+
+ operator = new AsyncWaitOperator<>(function);
+ Class<?>[] classes = AbstractStreamOperator.class.getDeclaredClasses();
+ Class<?> latencyClass = null;
+ for (Class<?> c : classes) {
+ if (c.getName().indexOf("LatencyGauge") != -1) {
+ latencyClass = c;
+ }
+ }
+
+ Constructor<?> explicitConstructor = latencyClass.getDeclaredConstructors()[0];
+ explicitConstructor.setAccessible(true);
+ Whitebox.setInternalState(operator, "latencyGauge", explicitConstructor.newInstance(10));
+
+ output = new FakedOutput(new ArrayList<Long>());
+ collector =new TimestampedCollector(output);
+
+ Whitebox.setInternalState(operator, "output", output);
+ }
+
+ @Test
+ public void testAdd() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+
+ buffer.addWatermark(new Watermark(0l));
+ buffer.addLatencyMarker(new LatencyMarker(111L, 1, 1));
+ Assert.assertEquals(buffer.getQueue().size(), 2);
+
+ Iterator<Map.Entry<AsyncCollector<Integer, Integer>, StreamElement>> iterator =
+ buffer.getQueue().entrySet().iterator();
+ Watermark watermark = iterator.next().getValue().asWatermark();
+ Assert.assertEquals(watermark.getTimestamp(), 0l);
+
+ LatencyMarker latencyMarker = iterator.next().getValue().asLatencyMarker();
+ Assert.assertEquals(latencyMarker.getMarkedTime(), 111l);
+
+ List<StreamElement> elements = buffer.getStreamElementsInBuffer();
+ Assert.assertEquals(elements.size(), 2);
+ }
+
+ public class FakedOutput implements Output<StreamRecord<Integer>> {
+ private Collection<Long> outputs;
+ private StringBuilder sb = new StringBuilder();
+
+ public FakedOutput(Collection<Long> outputs) {
+ this.outputs = outputs;
+ }
+
+ @Override
+ public void collect(StreamRecord<Integer> record) {
+ outputs.add(record.getValue().longValue());
+ if (record.hasTimestamp()) {
+ sb.append(record.getTimestamp() + ",");
+ }
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ outputs.add(mark.getTimestamp());
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ outputs.add(latencyMarker.getMarkedTime());
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public String getResult() {
+ StringBuilder sb = new StringBuilder();
+ for (Long i : outputs) {
+ sb.append(i).append(",");
+ }
+ return sb.toString();
+ }
+
+ public Collection<Long> getOutputs() {
+ return outputs;
+ }
+
+ public String getTimestamp() {
+ return this.sb.toString();
+ }
+ }
+
+ private void work(final boolean throwExcept) throws Exception {
+ final int ASYNC_COLLECTOR_NUM = 7;
+
+ final int[] orderedSeq = new int[] {0, 1, 2, 3, 4, 5, 6};
+ final int[] sleepTimeArr = new int[] {5, 7, 3, 0, 1, 9, 9};
+
+ ExecutorService service = Executors.newFixedThreadPool(10);
+
+ for (int idx = 0; idx < ASYNC_COLLECTOR_NUM; ++idx) {
+ int i = orderedSeq[idx];
+ final int sleepTS = sleepTimeArr[idx]*100;
+
+ if (i == 3) {
+ synchronized (lock) {
+ buffer.addWatermark(new Watermark(333l));
+ }
+ }
+ else if (i == 6) {
+ synchronized (lock) {
+ buffer.addLatencyMarker(new LatencyMarker(111L, 0, 0));
+ }
+ }
+ else {
+ StreamRecord record = new StreamRecord(i);
+ record.setTimestamp(i*i);
+
+ AsyncCollector tmp;
+ synchronized (lock) {
+ tmp = buffer.addStreamRecord(record);
+ }
+
+ final AsyncCollector collector = tmp;
+
+ final int v = i;
+ service.submit(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ int sleep = sleepTS;
+ Thread.sleep(sleep);
+
+ if (throwExcept) {
+ collector.collect(new Exception("wahahahaha..."));
+ }
+ else {
+ List<Integer> ret = new ArrayList<Integer>();
+
+ ret.add(v);
+ collector.collect(ret);
+ }
+ } catch (InterruptedException e) {
+ }
+ }
+ });
+ }
+ }
+
+ synchronized (lock) {
+ buffer.waitEmpty();
+ }
+ }
+
+ @Test
+ public void testOrderedBuffer() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.ORDERED, output, collector, lock, operator);
+ buffer.startEmitterThread();
+
+ work(false);
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+
+ Assert.assertEquals(((FakedOutput)output).getResult(), "0,1,2,333,4,5,111,");
+ Assert.assertEquals(((FakedOutput)output).getTimestamp(), "0,1,4,16,25,");
+ }
+
+ private String sort(String s) {
+ String[] arr = s.split(",");
+ int[] ints = new int[arr.length];
+ for (int i = 0; i < arr.length; ++i) {
+ ints[i] = Integer.valueOf(arr[i]);
+ }
+ Arrays.sort(ints);
+ return Arrays.toString(ints);
+ }
+
+ @Test
+ public void testUnorderedBuffer() throws Exception {
+ buffer =
+ new AsyncCollectorBuffer<>(3, AsyncDataStream.OutputMode.UNORDERED, output, collector, lock, operator);
+ buffer.startEmitterThread();
+
+ work(false);
+
+ buffer.waitEmpty();
+ buffer.stopEmitterThread();
+
+ Assert.assertEquals(((FakedOutput)output).getOutputs().toArray()[3], 333L);
+ Assert.assertEquals(sort(((FakedOutput)output).getResult()), "[0, 1, 2, 4, 5, 111, 333]");
+ Assert.assertEquals(sort(((FakedOutput)output).getTimestamp()), "[0, 1, 4, 16, 25]");
+ }
--- End diff --
I think these tests could be reworked to test more explicitly different interleavings of finished futures. So for example, the tests don't check:
- that a finished element but not the first but before the first watermark is emitted if it is completed
- that a finished element which is after the first watermark is not emitted
- Handling of special markers
- etc.
I think for that you should trigger the completion of the individual futures manually instead of relying on some sleep times. That way you have full control over when the elements are completed. This also applies to the ordered test case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86356593
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
--- End diff --
I think this type could be a `Queue`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2629: [FLINK-4391] Provide support for asynchronous operations ...
Posted by bjlovegithub <gi...@git.apache.org>.
Github user bjlovegithub commented on the issue:
https://github.com/apache/flink/pull/2629
Hi @tillrohrmann Thanks for your help. I am still working on it, adding more test cases and modifying its internal data structure. It almost has been done. But due to other jobs, I can not work on it in these days. I think I can update the PR probably on Wednesday after fixing UT failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r88869857
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * Keep all {@code AsyncCollector} and their input {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> queue = new LinkedHashMap<>();
+ /**
+ * For the AsyncWaitOperator chained with StreamSource, the checkpoint thread may get the
+ * {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock} while {@link AsyncCollectorBuffer#queue}
+ * is full since main thread waits on this lock. The StreamElement in
+ * {@link AsyncWaitOperator#processElement(StreamRecord)} should be treated as a part of all StreamElements
+ * in its queue. It will be kept in the operator state while snapshotting.
+ */
+ private StreamElement extraStreamElement;
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private final Output<StreamRecord<OUT>> output;
+ private final TimestampedCollector<OUT> timestampedCollector;
+
+ /**
+ * Checkpoint lock from {@link org.apache.flink.streaming.runtime.tasks.StreamTask#lock}
+ */
+ private final Object lock;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private IOException error;
+
+ public AsyncCollectorBuffer(
+ int bufferSize,
+ AsyncDataStream.OutputMode mode,
+ Output<StreamRecord<OUT>> output,
+ TimestampedCollector<OUT> collector,
+ Object lock,
+ AsyncWaitOperator operator) {
+ Preconditions.checkArgument(bufferSize > 0, "Future buffer size should be greater than 0.");
+ Preconditions.checkNotNull(output, "Output should not be NULL.");
+ Preconditions.checkNotNull(collector, "TimestampedCollector should not be NULL.");
+ Preconditions.checkNotNull(lock, "Checkpoint lock should not be NULL.");
+ Preconditions.checkNotNull(operator, "Reference to AsyncWaitOperator should not be NULL.");
+
+ this.bufferSize = bufferSize;
+ this.mode = mode;
+ this.output = output;
+ this.timestampedCollector = collector;
+ this.operator = operator;
+ this.lock = lock;
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until a new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addStreamRecord(StreamRecord<IN> record) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = record;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ queue.put(collector, record);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addWatermark(Watermark watermark) throws InterruptedException, IOException {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will wait here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> addLatencyMarker(LatencyMarker latencyMarker) throws InterruptedException, IOException {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws InterruptedException, IOException {
+ while (queue.size() >= bufferSize) {
+ // hold the input StreamRecord until it is placed in the buffer
+ extraStreamElement = mark;
+
+ lock.wait();
+ }
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ queue.put(collector, mark);
+
+ extraStreamElement = null;
+
+ return collector;
+ }
+
+ /**
+ * Notify the emitter thread and main thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void markCollectorCompleted(AsyncCollector<IN, OUT> collector) {
+ synchronized (lock) {
+ collector.markDone();
+
+ // notify main thread to keep working
+ lock.notifyAll();
+ }
+ }
+
+ /**
+ * Caller will wait here if buffer is not empty, meaning that not all async i/o tasks have returned yet.
+ *
+ * @throws Exception InterruptedException or IOException from AsyncCollector.
+ */
+ void waitEmpty() throws InterruptedException, IOException {
+ while (queue.size() != 0) {
+ if (error != null) {
+ throw error;
+ }
+
+ lock.wait();
+ }
+ }
+
+ public void startEmitterThread() {
+ this.emitThread.start();
+ }
+
+ public void stopEmitterThread() {
+ emitter.stop();
+
+ emitThread.interrupt();
+ }
+
+ /**
+ * Get all StreamElements in the AsyncCollector queue.
+ * <p>
+ * Emitter Thread can not output records and will wait for a while due to checkpoiting procedure
+ * holding the checkpoint lock.
+ *
+ * @return A List containing StreamElements.
+ */
+ public List<StreamElement> getStreamElementsInBuffer() {
+ List<StreamElement> ret = new ArrayList<>(queue.size());
+ for (Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry : queue.entrySet()) {
+ ret.add(entry.getValue());
+ }
+
+ // add the lonely input outside of queue into return set.
+ if (extraStreamElement != null) {
+ ret.add(extraStreamElement);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ public Map<AsyncCollector<IN, OUT>, StreamElement> getQueue() {
+ return this.queue;
+ }
+
+ /**
+ * A working thread to output results from {@link AsyncCollector} to the next operator.
+ */
+ private class Emitter implements Runnable {
+ private volatile boolean running = true;
+
+ private void output(AsyncCollector collector, StreamElement element) throws Exception {
+ List<OUT> result = collector.getResult();
+
+ // update timestamp for output stream records based on the input stream record.
+ if (element == null) {
+ throw new Exception("No input stream element for current AsyncCollector");
+ }
+
+ if (element.isRecord()) {
+ if (result == null) {
+ throw new Exception("Result for stream record "+element+" is null");
+ }
+
+ timestampedCollector.setTimestamp(element.asRecord());
+ for (OUT val : result) {
+ timestampedCollector.collect(val);
+ }
+ }
+ else if (element.isWatermark()) {
+ output.emitWatermark(element.asWatermark());
+ }
+ else if (element.isLatencyMarker()) {
+ operator.sendLatencyMarker(element.asLatencyMarker());
+ }
+ else {
+ throw new IOException("Unknown input record: "+element);
+ }
+ }
+
+ /**
+ * Emit results from the finished head collector and its following finished ones.
+ */
+ private void orderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ try {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (!collector.isDone()) {
+ break;
+ }
+
+ output(collector, entry.getValue());
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * Emit results for each finished collector. Try to emit results prior to the oldest watermark
+ * in the queue.
+ * <p>
+ * For example, assume the queue layout is:
+ * Entry(ac1, record1) -> Entry(ac2, record2) -> Entry(ac3, watermark1) -> Entry(ac4, record3).
+ * and both of ac2 and ac3 have finished. For unordered-mode, ac1 and ac2 are prior to watermark1,
+ * so ac2 will be emitted. Since ac1 is not ready yet, ac3 have to wait until ac1 is done.
+ */
+ private void unorderedProcess() throws IOException {
+ Iterator<Map.Entry<AsyncCollector<IN, OUT>, StreamElement>> iterator = queue.entrySet().iterator();
+
+ boolean hasUnfinishedCollector = false;
+
+ while (iterator.hasNext()) {
+ Map.Entry<AsyncCollector<IN, OUT>, StreamElement> entry = iterator.next();
+
+ StreamElement element = entry.getValue();
+ AsyncCollector<IN, OUT> collector = entry.getKey();
+
+ if (element.isWatermark() && hasUnfinishedCollector) {
+ break;
+ }
+
+ if (collector.isDone() == false) {
+ hasUnfinishedCollector = true;
+
+ continue;
+ }
+
+ try {
+ output(collector, element);
+
+ iterator.remove();
+
+ lock.notifyAll();
+ }
+ catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ /**
+ * If
+ * In ordered mode, there are some finished async collectors, and one of them is the first element in
+ * the queue.
+ * or
+ * In unordered mode, there are some finished async collectors prior to the oldest water mark.
+ * or
+ * The first element in the queue is Watermark or LatencyMarker.
+ * Then, the emitter thread should keep waiting, rather than waiting on the condition.
+ *
+ * Otherwise, the thread should stop for a while until being signalled.
+ */
+ private boolean nothingToDo() {
--- End diff --
Then we could also do in the Emitter's run method a little bit more fine-grained locking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r86355297
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollectorBuffer.java ---
@@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * AsyncCollectorBuffer will hold all {@link AsyncCollector} in its internal buffer,
+ * and emit results from {@link AsyncCollector} to the next operators following it by
+ * calling {@link Output#collect(Object)}
+ */
+@Internal
+public class AsyncCollectorBuffer<IN, OUT> {
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncCollectorBuffer.class);
+
+ /**
+ * Max number of {@link AsyncCollector} in the buffer.
+ */
+ private final int bufferSize;
+
+ private final AsyncDataStream.OutputMode mode;
+
+ private final AsyncWaitOperator<IN, OUT> operator;
+
+ /**
+ * {@link AsyncCollector} queue.
+ */
+ private final SimpleLinkedList<AsyncCollector<IN, OUT>> queue = new SimpleLinkedList<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their corresponding {@link StreamElement}
+ */
+ private final Map<AsyncCollector<IN, OUT>, StreamElement> collectorToStreamElement = new HashMap<>();
+ /**
+ * A hash map keeping {@link AsyncCollector} and their node references in the #queue.
+ */
+ private final Map<AsyncCollector<IN, OUT>, SimpleLinkedList.Node> collectorToQueue = new HashMap<>();
+
+ private final LinkedList<AsyncCollector> finishedCollectors = new LinkedList<>();
+
+ /**
+ * {@link TimestampedCollector} and {@link Output} to collect results and watermarks.
+ */
+ private TimestampedCollector<OUT> timestampedCollector;
+ private Output<StreamRecord<OUT>> output;
+
+ /**
+ * Locks and conditions to synchronize with main thread and emitter thread.
+ */
+ private final Lock lock;
+ private final Condition notFull;
+ private final Condition taskDone;
+ private final Condition isEmpty;
+
+ /**
+ * Error from user codes.
+ */
+ private volatile Exception error;
+
+ private final Emitter emitter;
+ private final Thread emitThread;
+
+ private boolean isCheckpointing;
+
+ public AsyncCollectorBuffer(int maxSize, AsyncDataStream.OutputMode mode, AsyncWaitOperator operator) {
+ Preconditions.checkArgument(maxSize > 0, "Future buffer size should be greater than 0.");
+
+ this.bufferSize = maxSize;
+ this.mode = mode;
+ this.operator = operator;
+
+ this.lock = new ReentrantLock(true);
+ this.notFull = this.lock.newCondition();
+ this.taskDone = this.lock.newCondition();
+ this.isEmpty = this.lock.newCondition();
+
+ this.emitter = new Emitter();
+ this.emitThread = new Thread(emitter);
+ }
+
+ /**
+ * Add an {@link StreamRecord} into the buffer. A new {@link AsyncCollector} will be created and returned
+ * corresponding to the input StreamRecord.
+ * <p>
+ * If buffer is full, caller will wait until new space is available.
+ *
+ * @param record StreamRecord
+ * @return An AsyncCollector
+ * @throws Exception InterruptedException or exceptions from AsyncCollector.
+ */
+ public AsyncCollector<IN, OUT> add(StreamRecord<IN> record) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize) {
+ notFull.await();
+ }
+
+ // propagate error to the main thread
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, record);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Add a {@link Watermark} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param watermark Watermark
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(Watermark watermark) throws Exception {
+ return processMark(watermark);
+ }
+
+ /**
+ * Add a {@link LatencyMarker} into queue. A new AsyncCollector will be created and returned.
+ * <p>
+ * If queue is full, caller will be blocked here.
+ *
+ * @param latencyMarker LatencyMarker
+ * @return AsyncCollector
+ * @throws Exception Exceptions from async operation.
+ */
+ public AsyncCollector<IN, OUT> add(LatencyMarker latencyMarker) throws Exception {
+ return processMark(latencyMarker);
+ }
+
+ private AsyncCollector<IN, OUT> processMark(StreamElement mark) throws Exception {
+ try {
+ lock.lock();
+
+ notifyCheckpointDone();
+
+ while (queue.size() >= bufferSize)
+ notFull.await();
+
+ if (error != null) {
+ throw error;
+ }
+
+ AsyncCollector<IN, OUT> collector = new AsyncCollector(this, true);
+
+ collectorToQueue.put(collector, queue.add(collector));
+ collectorToStreamElement.put(collector, mark);
+
+ // signal emitter thread that current collector is ready
+ mark(collector);
+
+ return collector;
+ }
+ finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Notify the Emitter Thread that an AsyncCollector has completed.
+ *
+ * @param collector Completed AsyncCollector
+ */
+ void mark(AsyncCollector<IN, OUT> collector) {
--- End diff --
I think this method should get a more expressive name. Something like `markCollectorCompleted` or `completeCollector` would be a little bit better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---