You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/19 05:52:30 UTC
[GitHub] [spark] dchvn opened a new pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
dchvn opened a new pull request #34323:
URL: https://github.com/apache/spark/pull/34323
### What changes were proposed in this pull request?
Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
### Why are the changes needed?
We can take advantage of static type checking within the functions by inlining the type hints.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951514571
**[Test build #144606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144606/testReport)** for PR 34323 at commit [`4684e6c`](https://github.com/apache/spark/commit/4684e6c2313f46e5ecf2a76e900f743c3f235520).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952807016
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951562785
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r734152016
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -34,11 +38,22 @@ def utf8_decoder(s):
class KinesisUtils(object):
@staticmethod
- def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
- initialPositionInStream, checkpointInterval,
- storageLevel=StorageLevel.MEMORY_AND_DISK_2,
- awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
- stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
+ def createStreamcreateStream(
Review comment:
Sorry my bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-954040786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49191/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953994734
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144721/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949239343
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48995/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951537319
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144606/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r738217455
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -47,10 +47,10 @@ def createStream(
regionName: str,
initialPositionInStream: str,
checkpointInterval: int,
+ *,
Review comment:
Sorry, updated. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946446582
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r738160026
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -47,10 +47,10 @@ def createStream(
regionName: str,
initialPositionInStream: str,
checkpointInterval: int,
+ *,
Review comment:
Sorry,, but we're still not there.
Any argument positioned before `decoder` can be used in both keyword and positional. Only when you omit `decoder`, remaining arguments have to be named to avoid ambiguity. So `*` should be exactly where `decoder` was.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953507553
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949262280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48995/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953466122
**[Test build #144691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144691/testReport)** for PR 34323 at commit [`3ecf4ff`](https://github.com/apache/spark/commit/3ecf4ff4668956bcf332d17512b724810b53d5f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946408617
**[Test build #144398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144398/testReport)** for PR 34323 at commit [`f83a44f`](https://github.com/apache/spark/commit/f83a44f406eccdc300c4c375c6e9af743e74195d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949262280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48995/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953744680
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953744680
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953741318
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49181/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r737292998
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -34,11 +38,65 @@ def utf8_decoder(s):
class KinesisUtils(object):
@staticmethod
- def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
- initialPositionInStream, checkpointInterval,
- storageLevel=StorageLevel.MEMORY_AND_DISK_2,
- awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
- stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
+ @overload
+ def createStream(
+ ssc: StreamingContext,
+ kinesisAppName: str,
+ streamName: str,
+ endpointUrl: str,
+ regionName: str,
+ initialPositionInStream: str,
+ checkpointInterval: int,
+ storageLevel: StorageLevel = ...,
+ awsAccessKeyId: Optional[str] = ...,
+ awsSecretKey: Optional[str] = ...,
+ decoder: Callable[[Optional[bytes]], Optional[str]] = ...,
Review comment:
W don't need `decoder` here. Instead, all remaining arguments should be keyword-only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951537319
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144606/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953964346
**[Test build #144721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144721/testReport)** for PR 34323 at commit [`deee5d2`](https://github.com/apache/spark/commit/deee5d21107ca4817c781be871839c09b4852f79).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953944361
**[Test build #144721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144721/testReport)** for PR 34323 at commit [`deee5d2`](https://github.com/apache/spark/commit/deee5d21107ca4817c781be871839c09b4852f79).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946421708
**[Test build #144398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144398/testReport)** for PR 34323 at commit [`f83a44f`](https://github.com/apache/spark/commit/f83a44f406eccdc300c4c375c6e9af743e74195d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946443514
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144398/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r737989238
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -34,11 +38,65 @@ def utf8_decoder(s):
class KinesisUtils(object):
@staticmethod
- def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
- initialPositionInStream, checkpointInterval,
- storageLevel=StorageLevel.MEMORY_AND_DISK_2,
- awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
- stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
+ @overload
+ def createStream(
+ ssc: StreamingContext,
+ kinesisAppName: str,
+ streamName: str,
+ endpointUrl: str,
+ regionName: str,
+ initialPositionInStream: str,
+ checkpointInterval: int,
+ storageLevel: StorageLevel = ...,
+ awsAccessKeyId: Optional[str] = ...,
+ awsSecretKey: Optional[str] = ...,
+ decoder: Callable[[Optional[bytes]], Optional[str]] = ...,
Review comment:
Thanks! Could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-954443411
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946443514
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144398/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r734210281
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -34,11 +38,22 @@ def utf8_decoder(s):
class KinesisUtils(object):
@staticmethod
- def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
- initialPositionInStream, checkpointInterval,
- storageLevel=StorageLevel.MEMORY_AND_DISK_2,
- awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
- stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
+ def createStreamcreateStream(
Review comment:
Updated, thank you !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951553058
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r738177769
##########
File path: python/pyspark/streaming/listener.py
##########
@@ -15,63 +15,66 @@
# limitations under the License.
#
+from typing import Any
+
+
__all__ = ["StreamingListener"]
class StreamingListener(object):
- def __init__(self):
+ def __init__(self) -> None:
Review comment:
Nit. We don't need return annotation on initializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953707606
**[Test build #144712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144712/testReport)** for PR 34323 at commit [`dd744be`](https://github.com/apache/spark/commit/dd744be7d2870f191b065856a9b62cbaea8a9129).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951514571
**[Test build #144606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144606/testReport)** for PR 34323 at commit [`4684e6c`](https://github.com/apache/spark/commit/4684e6c2313f46e5ecf2a76e900f743c3f235520).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953989316
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49191/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953484155
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952757436
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144655/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r738475547
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -14,17 +14,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from typing import TypeVar, Optional, Callable, Union, overload
Review comment:
Thanks! updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952757436
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144655/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953944361
**[Test build #144721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144721/testReport)** for PR 34323 at commit [`deee5d2`](https://github.com/apache/spark/commit/deee5d21107ca4817c781be871839c09b4852f79).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949255321
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48995/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r736118182
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -14,17 +14,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from typing import TypeVar, Optional, Callable
from pyspark.serializers import NoOpSerializer
from pyspark.storagelevel import StorageLevel
from pyspark.streaming import DStream
-from pyspark.util import _print_missing_jar
+from pyspark.streaming.context import StreamingContext
+from pyspark.util import _print_missing_jar # type: ignore[attr-defined]
__all__ = ['KinesisUtils', 'InitialPositionInStream', 'utf8_decoder']
+T = TypeVar("T")
Review comment:
I replaced ```ignore``` by ```cast```, could you take another look ? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953507553
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946410931
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946446550
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949153768
Looks fine otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949242139
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144524/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949192690
**[Test build #144524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144524/testReport)** for PR 34323 at commit [`93da9be`](https://github.com/apache/spark/commit/93da9be8df0dea11c699033a740cd13fc7722455).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r734239196
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -14,17 +14,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from typing import TypeVar, Optional, Callable
from pyspark.serializers import NoOpSerializer
from pyspark.storagelevel import StorageLevel
from pyspark.streaming import DStream
-from pyspark.util import _print_missing_jar
+from pyspark.streaming.context import StreamingContext
+from pyspark.util import _print_missing_jar # type: ignore[attr-defined]
__all__ = ['KinesisUtils', 'InitialPositionInStream', 'utf8_decoder']
+T = TypeVar("T")
Review comment:
Can you see if it works if you set `covariant=True`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949242139
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144524/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r734229556
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -34,11 +38,22 @@ def utf8_decoder(s):
class KinesisUtils(object):
@staticmethod
- def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
- initialPositionInStream, checkpointInterval,
- storageLevel=StorageLevel.MEMORY_AND_DISK_2,
- awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
- stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
+ def createStream(
+ ssc: StreamingContext,
+ kinesisAppName: str,
+ streamName: str,
+ endpointUrl: str,
+ regionName: str,
+ initialPositionInStream: str,
+ checkpointInterval: int,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2,
+ awsAccessKeyId: Optional[str] = None,
+ awsSecretKey: Optional[str] = None,
+ decoder: Callable[[Optional[bytes]], T] = utf8_decoder, # type: ignore[assignment]
Review comment:
Oh, because we have
```utf8_decoder: Callable[[Optional[bytes]], Optional[str]]``` but ```decoder``` is ```Callable[[Optional[bytes]], T]```.
Do you have any suggestion? I am facing with many errors like that in ```python/pyspark/rdd.py```. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r734399027
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -14,17 +14,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from typing import TypeVar, Optional, Callable
from pyspark.serializers import NoOpSerializer
from pyspark.storagelevel import StorageLevel
from pyspark.streaming import DStream
-from pyspark.util import _print_missing_jar
+from pyspark.streaming.context import StreamingContext
+from pyspark.util import _print_missing_jar # type: ignore[attr-defined]
__all__ = ['KinesisUtils', 'InitialPositionInStream', 'utf8_decoder']
+T = TypeVar("T")
Review comment:
thanks, it seem doesn't work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953485654
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144691/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951534980
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953505980
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952738824
**[Test build #144655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144655/testReport)** for PR 34323 at commit [`21206fe`](https://github.com/apache/spark/commit/21206fec91cde616e69dd75e23858305b5d80844).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953785176
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49181/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953994734
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144721/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952807016
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r734139380
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -34,11 +38,22 @@ def utf8_decoder(s):
class KinesisUtils(object):
@staticmethod
- def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
- initialPositionInStream, checkpointInterval,
- storageLevel=StorageLevel.MEMORY_AND_DISK_2,
- awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
- stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
+ def createStreamcreateStream(
Review comment:
Why was the name changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946408617
**[Test build #144398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144398/testReport)** for PR 34323 at commit [`f83a44f`](https://github.com/apache/spark/commit/f83a44f406eccdc300c4c375c6e9af743e74195d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952757321
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953474581
**[Test build #144691 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144691/testReport)** for PR 34323 at commit [`3ecf4ff`](https://github.com/apache/spark/commit/3ecf4ff4668956bcf332d17512b724810b53d5f1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953485654
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144691/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951562785
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952719129
**[Test build #144655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144655/testReport)** for PR 34323 at commit [`21206fe`](https://github.com/apache/spark/commit/21206fec91cde616e69dd75e23858305b5d80844).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953725760
**[Test build #144712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144712/testReport)** for PR 34323 at commit [`dd744be`](https://github.com/apache/spark/commit/dd744be7d2870f191b065856a9b62cbaea8a9129).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952792927
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r738197128
##########
File path: python/pyspark/streaming/listener.py
##########
@@ -15,63 +15,66 @@
# limitations under the License.
#
+from typing import Any
+
+
__all__ = ["StreamingListener"]
class StreamingListener(object):
- def __init__(self):
+ def __init__(self) -> None:
Review comment:
Could you take a look at this https://mypy.readthedocs.io/en/stable/error_code_list2.html
It have an example:
``` python
class Counter:
# Error: Function is missing a type annotation [no-untyped-def]
def __init__(self):
self.value = 0
class CounterOk:
# OK: An explicit "-> None" is needed if "__init__" takes no arguments
def __init__(self) -> None:
self.value = 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-954432054
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-954035913
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49191/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-954040786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49191/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34323:
URL: https://github.com/apache/spark/pull/34323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953466122
**[Test build #144691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144691/testReport)** for PR 34323 at commit [`3ecf4ff`](https://github.com/apache/spark/commit/3ecf4ff4668956bcf332d17512b724810b53d5f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r738404483
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -14,17 +14,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from typing import TypeVar, Optional, Callable, Union, overload
Review comment:
Nit. Let's sort these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953777911
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49181/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953785176
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49181/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-953707606
**[Test build #144712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144712/testReport)** for PR 34323 at commit [`dd744be`](https://github.com/apache/spark/commit/dd744be7d2870f191b065856a9b62cbaea8a9129).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r736257476
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -14,17 +14,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from typing import TypeVar, Optional, Callable
from pyspark.serializers import NoOpSerializer
from pyspark.storagelevel import StorageLevel
from pyspark.streaming import DStream
-from pyspark.util import _print_missing_jar
+from pyspark.streaming.context import StreamingContext
+from pyspark.util import _print_missing_jar # type: ignore[attr-defined]
__all__ = ['KinesisUtils', 'InitialPositionInStream', 'utf8_decoder']
+T = TypeVar("T")
Review comment:
This should be probably overloaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-952719129
**[Test build #144655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144655/testReport)** for PR 34323 at commit [`21206fe`](https://github.com/apache/spark/commit/21206fec91cde616e69dd75e23858305b5d80844).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-946446582
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48870/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-948179912
CC @HyukjinKwon @ueshin @zero323 FYI. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34323:
URL: https://github.com/apache/spark/pull/34323#discussion_r734226805
##########
File path: python/pyspark/streaming/kinesis.py
##########
@@ -34,11 +38,22 @@ def utf8_decoder(s):
class KinesisUtils(object):
@staticmethod
- def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
- initialPositionInStream, checkpointInterval,
- storageLevel=StorageLevel.MEMORY_AND_DISK_2,
- awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
- stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
+ def createStream(
+ ssc: StreamingContext,
+ kinesisAppName: str,
+ streamName: str,
+ endpointUrl: str,
+ regionName: str,
+ initialPositionInStream: str,
+ checkpointInterval: int,
+ storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_2,
+ awsAccessKeyId: Optional[str] = None,
+ awsSecretKey: Optional[str] = None,
+ decoder: Callable[[Optional[bytes]], T] = utf8_decoder, # type: ignore[assignment]
Review comment:
why does it need `ignore[assignment]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949192690
**[Test build #144524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144524/testReport)** for PR 34323 at commit [`93da9be`](https://github.com/apache/spark/commit/93da9be8df0dea11c699033a740cd13fc7722455).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-949221254
**[Test build #144524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144524/testReport)** for PR 34323 at commit [`93da9be`](https://github.com/apache/spark/commit/93da9be8df0dea11c699033a740cd13fc7722455).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34323: [SPARK-37042][PYTHON] Inline type hints for kinesis.py and listener.py in python/pyspark/streaming
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34323:
URL: https://github.com/apache/spark/pull/34323#issuecomment-951524936
**[Test build #144606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144606/testReport)** for PR 34323 at commit [`4684e6c`](https://github.com/apache/spark/commit/4684e6c2313f46e5ecf2a76e900f743c3f235520).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org