You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Noam Tamim (Jira)" <ji...@apache.org> on 2021/08/16 08:38:00 UTC
[jira] [Updated] (SPARK-36520) KinesisInitialPosition interface
should be public
[ https://issues.apache.org/jira/browse/SPARK-36520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noam Tamim updated SPARK-36520:
-------------------------------
Description:
In `org.apache.spark.streaming.kinesis` there's a Java interface `KinesisInitialPosition` - for some reason it's not public, so can't be used.
The reason it needs to be used is simple: it is the parent of `KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`. Without using it, it's impossible to pass instances of `Latest` and `TrimHorizon` around without calling them by name.
The workaround is to use the enum `InitialPositionInStream` instead, and a factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like the correct pattern, because it belongs to Amazon Kinesis Client library and not Spark (it's currently the only import in my class which is outside of `org.apache.spark`).
The change is trivial, so it's mainly a matter of why *not* to do it.
KinesisInitialPositions: https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/java/org/apache/spark/streaming/kinesis/KinesisInitialPositions.java
KinesisInitialPosition: https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java
was:
In `org.apache.spark.streaming.kinesis` there's a Java interface `KinesisInitialPosition` - for some reason it's not public, so can't be used.
The reason it needs to be used is simple: it is the parent of `KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`. Without using it, it's impossible to pass instances of `Latest` and `TrimHorizon` around without calling them by name.
The workaround is to use the enum `InitialPositionInStream` instead, and a factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like the correct pattern.
> KinesisInitialPosition interface should be public
> -------------------------------------------------
>
> Key: SPARK-36520
> URL: https://issues.apache.org/jira/browse/SPARK-36520
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 3.1.2
> Reporter: Noam Tamim
> Priority: Minor
> Labels: kinesis
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> In `org.apache.spark.streaming.kinesis` there's a Java interface `KinesisInitialPosition` - for some reason it's not public, so can't be used.
> The reason it needs to be used is simple: it is the parent of `KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`. Without using it, it's impossible to pass instances of `Latest` and `TrimHorizon` around without calling them by name.
> The workaround is to use the enum `InitialPositionInStream` instead, and a factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like the correct pattern, because it belongs to Amazon Kinesis Client library and not Spark (it's currently the only import in my class which is outside of `org.apache.spark`).
> The change is trivial, so it's mainly a matter of why *not* to do it.
> KinesisInitialPositions: https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/java/org/apache/spark/streaming/kinesis/KinesisInitialPositions.java
> KinesisInitialPosition: https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org