You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/05/06 14:57:05 UTC
[GitHub] [beam] j-arrow opened a new pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
j-arrow opened a new pull request #14743:
URL: https://github.com/apache/beam/pull/14743
Beam currently uses AWS DescribeStream API to list Kinesis shards. There are few problems with it:
- it is deprecated as per AWS [documentation](https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-list-shards.html)
- it is slow, as it requires additional validation of listed shards (it returns all of them, including closed ones, so GetShardIterator calls are required)
- it has low transaction limit per second (per account)
- AWS changed the behaviour of GetShardIterator (it returns iterator for closed shards) and because of that Beam might read too old data, leading to increase in costs and delay
Use of ListShards API eliminates problems mentioned above.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r645469349
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
##########
@@ -61,18 +69,33 @@
private static final int PERIOD_GRANULARITY_IN_SECONDS = 60;
private static final String SUM_STATISTIC = "Sum";
private static final String STREAM_NAME_DIMENSION = "StreamName";
- private static final int LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS = 10;
- private static final Duration LIST_SHARDS_DESCRIBE_STREAM_INITIAL_BACKOFF =
+ private static final int LIST_SHARDS_MAX_RESULTS = 1_000;
+ private static final Duration
+ SPACING_FOR_TIMESTAMP_LIST_SHARDS_REQUEST_TO_NOT_EXCEED_TRIM_HORIZON =
+ Duration.standardMinutes(5);
+ private static final int DESCRIBE_STREAM_SUMMARY_MAX_ATTEMPTS = 10;
+ private static final Duration DESCRIBE_STREAM_SUMMARY_INITIAL_BACKOFF =
Duration.standardSeconds(1);
+
private final AmazonKinesis kinesis;
private final AmazonCloudWatch cloudWatch;
private final Integer limit;
+ private final Supplier<Instant> currentInstantSupplier;
public SimplifiedKinesisClient(
AmazonKinesis kinesis, AmazonCloudWatch cloudWatch, Integer limit) {
+ this(kinesis, cloudWatch, limit, Instant::now);
Review comment:
This class is package-private, so even if constructor is public it should not be a problem. Constructor doesn't have new parameter, it provides default value. `checkNotNull` calls verifying arguments that were here before are in package-private constructor used for tests.
Could you please pinpoint where API has changed, @aromanenko-dev ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-842216433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r636761081
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
##########
@@ -120,6 +122,13 @@ void start() throws TransientKinesisException {
// Note: readLoop() will log any Throwable raised so opt to ignore the future result
@SuppressWarnings("FutureReturnValueIgnored")
void startReadingShards(Iterable<ShardRecordsIterator> shardRecordsIterators) {
+ if (!shardRecordsIterators.iterator().hasNext()) {
+ return;
Review comment:
Maybe add a log message about this?
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
##########
@@ -61,18 +69,33 @@
private static final int PERIOD_GRANULARITY_IN_SECONDS = 60;
private static final String SUM_STATISTIC = "Sum";
private static final String STREAM_NAME_DIMENSION = "StreamName";
- private static final int LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS = 10;
- private static final Duration LIST_SHARDS_DESCRIBE_STREAM_INITIAL_BACKOFF =
+ private static final int LIST_SHARDS_MAX_RESULTS = 1_000;
+ private static final Duration
+ SPACING_FOR_TIMESTAMP_LIST_SHARDS_REQUEST_TO_NOT_EXCEED_TRIM_HORIZON =
+ Duration.standardMinutes(5);
+ private static final int DESCRIBE_STREAM_SUMMARY_MAX_ATTEMPTS = 10;
+ private static final Duration DESCRIBE_STREAM_SUMMARY_INITIAL_BACKOFF =
Duration.standardSeconds(1);
+
private final AmazonKinesis kinesis;
private final AmazonCloudWatch cloudWatch;
private final Integer limit;
+ private final Supplier<Instant> currentInstantSupplier;
public SimplifiedKinesisClient(
AmazonKinesis kinesis, AmazonCloudWatch cloudWatch, Integer limit) {
+ this(kinesis, cloudWatch, limit, Instant::now);
Review comment:
Isn't this a part of public user API that is being changed?
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java
##########
@@ -46,19 +45,19 @@ public StartingPoint(InitialPositionInStream position) {
public StartingPoint(Instant timestamp) {
this.timestamp = checkNotNull(timestamp, "timestamp");
- this.position = null;
+ this.position = InitialPositionInStream.AT_TIMESTAMP;
Review comment:
Why it was changed?
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
##########
@@ -61,18 +69,33 @@
private static final int PERIOD_GRANULARITY_IN_SECONDS = 60;
private static final String SUM_STATISTIC = "Sum";
private static final String STREAM_NAME_DIMENSION = "StreamName";
- private static final int LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS = 10;
- private static final Duration LIST_SHARDS_DESCRIBE_STREAM_INITIAL_BACKOFF =
+ private static final int LIST_SHARDS_MAX_RESULTS = 1_000;
+ private static final Duration
+ SPACING_FOR_TIMESTAMP_LIST_SHARDS_REQUEST_TO_NOT_EXCEED_TRIM_HORIZON =
Review comment:
Can we name it with a shorter name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-842216433
@j-arrow Thanks! I'll take a look asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #14743:
URL: https://github.com/apache/beam/pull/14743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-839742018
Hi @aromanenko-dev. I have fixed the compilation errors that were detected by the pipeline. Will you be able to take a look at these changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r645422497
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java
##########
@@ -46,19 +45,19 @@ public StartingPoint(InitialPositionInStream position) {
public StartingPoint(Instant timestamp) {
this.timestamp = checkNotNull(timestamp, "timestamp");
- this.position = null;
+ this.position = InitialPositionInStream.AT_TIMESTAMP;
Review comment:
Please take a look at for example [`org.apache.beam.sdk.io.kinesis.StartingPoint#getPositionName`](https://github.com/apache/beam/blob/d20e063bcd683880e311fd06430e6e2a0dd90937/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java#L57). It had an `if` statement that could be replaced with setting `position` to `AT_TIMESTAMP` value in this constructor.
Another issue would be its use in [`org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient#buildShardFilterForStartingPoint`](https://github.com/apache/beam/blob/d20e063bcd683880e311fd06430e6e2a0dd90937/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java#L57) would require `null` check.
It eliminates these problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-833600787
R: @aromanenko-dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r645414935
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
##########
@@ -61,18 +69,33 @@
private static final int PERIOD_GRANULARITY_IN_SECONDS = 60;
private static final String SUM_STATISTIC = "Sum";
private static final String STREAM_NAME_DIMENSION = "StreamName";
- private static final int LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS = 10;
- private static final Duration LIST_SHARDS_DESCRIBE_STREAM_INITIAL_BACKOFF =
+ private static final int LIST_SHARDS_MAX_RESULTS = 1_000;
+ private static final Duration
+ SPACING_FOR_TIMESTAMP_LIST_SHARDS_REQUEST_TO_NOT_EXCEED_TRIM_HORIZON =
Review comment:
This name is so long to better describe what it does. if it would be shorter, it would be hard to describe what it does without adding comment to it. As such, I think it is better way to use the one I have proposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r645422497
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java
##########
@@ -46,19 +45,19 @@ public StartingPoint(InitialPositionInStream position) {
public StartingPoint(Instant timestamp) {
this.timestamp = checkNotNull(timestamp, "timestamp");
- this.position = null;
+ this.position = InitialPositionInStream.AT_TIMESTAMP;
Review comment:
Please take a look at for example [`org.apache.beam.sdk.io.kinesis.StartingPoint#getPositionName`](https://github.com/apache/beam/blob/d20e063bcd683880e311fd06430e6e2a0dd90937/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java#L57). It had an `if` statement that could be replaced with setting `position` to `AT_TIMESTAMP` value in this constructor.
Another issue would be its use in [`org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient#buildShardFilterForStartingPoint`](https://github.com/apache/beam/blob/42ada6201618b80a037269c81876a5f37eec5e6c/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java#L137) would require `null` check.
It eliminates these problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r645499525
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
##########
@@ -61,18 +69,33 @@
private static final int PERIOD_GRANULARITY_IN_SECONDS = 60;
private static final String SUM_STATISTIC = "Sum";
private static final String STREAM_NAME_DIMENSION = "StreamName";
- private static final int LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS = 10;
- private static final Duration LIST_SHARDS_DESCRIBE_STREAM_INITIAL_BACKOFF =
+ private static final int LIST_SHARDS_MAX_RESULTS = 1_000;
+ private static final Duration
+ SPACING_FOR_TIMESTAMP_LIST_SHARDS_REQUEST_TO_NOT_EXCEED_TRIM_HORIZON =
Review comment:
Personally, I'd prefer to have a shorter name with a comment, which describes it in more details, but it's ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-854103404
@j-arrow - Would you be able to respond to the open comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-854822964
@j-arrow Not a problem at all, thanks for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r645497538
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
##########
@@ -61,18 +69,33 @@
private static final int PERIOD_GRANULARITY_IN_SECONDS = 60;
private static final String SUM_STATISTIC = "Sum";
private static final String STREAM_NAME_DIMENSION = "StreamName";
- private static final int LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS = 10;
- private static final Duration LIST_SHARDS_DESCRIBE_STREAM_INITIAL_BACKOFF =
+ private static final int LIST_SHARDS_MAX_RESULTS = 1_000;
+ private static final Duration
+ SPACING_FOR_TIMESTAMP_LIST_SHARDS_REQUEST_TO_NOT_EXCEED_TRIM_HORIZON =
+ Duration.standardMinutes(5);
+ private static final int DESCRIBE_STREAM_SUMMARY_MAX_ATTEMPTS = 10;
+ private static final Duration DESCRIBE_STREAM_SUMMARY_INITIAL_BACKOFF =
Duration.standardSeconds(1);
+
private final AmazonKinesis kinesis;
private final AmazonCloudWatch cloudWatch;
private final Integer limit;
+ private final Supplier<Instant> currentInstantSupplier;
public SimplifiedKinesisClient(
AmazonKinesis kinesis, AmazonCloudWatch cloudWatch, Integer limit) {
+ this(kinesis, cloudWatch, limit, Instant::now);
Review comment:
You right, I was mistaken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-854103404
@j-arrow - Would you be able to respond to the open comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-854731881
Thank you for the review, @aromanenko-dev and sorry for monstrously prolonging this pull request!
Branch is rebased and commits are squashed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-854500870
On it right now, sorry for delay @aaltay .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-842463213
CC: @pawel-kaczmarczyk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] j-arrow commented on a change in pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
j-arrow commented on a change in pull request #14743:
URL: https://github.com/apache/beam/pull/14743#discussion_r645524248
##########
File path: sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
##########
@@ -120,6 +122,13 @@ void start() throws TransientKinesisException {
// Note: readLoop() will log any Throwable raised so opt to ignore the future result
@SuppressWarnings("FutureReturnValueIgnored")
void startReadingShards(Iterable<ShardRecordsIterator> shardRecordsIterators) {
+ if (!shardRecordsIterators.iterator().hasNext()) {
+ return;
Review comment:
I have added log message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #14743: [BEAM-12225] Replace AWS API used to list shards from DescribeStream to ListShards
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #14743:
URL: https://github.com/apache/beam/pull/14743#issuecomment-833603262
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org