You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "rangadi (via GitHub)" <gi...@apache.org> on 2023/10/18 05:54:12 UTC
[PR] [SPARK-45588]Minor Scaladoc improvement for StreamingForeachBatchHelper [spark]
rangadi opened a new pull request, #43424:
URL: https://github.com/apache/spark/pull/43424
### What changes were proposed in this pull request?
Couple of minor improvements to `StreamingForeachBatchHelper`:
* Make `RunnerCleaner` private and add ScalaDoc.
* Update contract for `pythonForeachBatchWrapper()` to inform that call should eventually should `close()` the `AutoClosable` returned.
### Why are the changes needed?
- Code readability improvement.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Existing tests. Minor change.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45588]Minor Scaladoc improvement for StreamingForeachBatchHelper [spark]
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on PR #43424:
URL: https://github.com/apache/spark/pull/43424#issuecomment-1767693728
cc: @WweiL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45588][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #43424: [SPARK-45588][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper
URL: https://github.com/apache/spark/pull/43424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45588][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #43424:
URL: https://github.com/apache/spark/pull/43424#discussion_r1364776476
##########
connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala:
##########
@@ -137,7 +137,7 @@ class ProtobufCatalystDataConversionSuite
while (
data != null &&
(data.get(0) == defaultValue ||
- (dt == BinaryType &&
+ (dt.fields(0).dataType == BinaryType &&
Review Comment:
Ah OK that's actually failing consistently for specific value but the test itself is using random seed hence flaky...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45588]Minor Scaladoc improvement for StreamingForeachBatchHelper [spark]
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #43424:
URL: https://github.com/apache/spark/pull/43424#discussion_r1364757563
##########
connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala:
##########
@@ -137,7 +137,7 @@ class ProtobufCatalystDataConversionSuite
while (
data != null &&
(data.get(0) == defaultValue ||
- (dt == BinaryType &&
+ (dt.fields(0).dataType == BinaryType &&
Review Comment:
This fixes flake reported in https://github.com/apache/spark/pull/43413#issuecomment-1769286752
The type check was incorrect. So when the random value was empty array, we didn't skip it. Original intention is to skip default values for types. Verified with manually setting the seed to 399 (at line 128).
cc: @HeartSaVioR
<img width="1480" alt="image" src="https://github.com/apache/spark/assets/502522/94a8aca2-328c-4371-9067-f0ff18d59574">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45588][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper [spark]
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on PR #43424:
URL: https://github.com/apache/spark/pull/43424#issuecomment-1769964614
@HeartSaVioR all the tests passes. Updated "testing" section in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45588][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #43424:
URL: https://github.com/apache/spark/pull/43424#issuecomment-1769982937
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45588]Minor Scaladoc improvement for StreamingForeachBatchHelper [spark]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #43424:
URL: https://github.com/apache/spark/pull/43424#issuecomment-1768503199
Looks OK, just get tests to pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org