[GitHub] [beam] lukecwik commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

[GitBox <gi...@apache.org>]

lukecwik commented on pull request #12603:
URL: https://github.com/apache/beam/pull/12603#issuecomment-694659139


   @iemejia I have updated the code and added a `SparkProcessedKeyedElements` using `updateStateByKey` to evaluate a splittable DoFn. I based the logic off of the `SparkGroupAlsoByWindowViaWindowSet` logic. I also added some `SplittableDoFnTest`s and have been using `org.apache.beam.runners.spark.translation.streaming.SplittableDoFnTest#testPairWithIndexBasicUnbounded` as my base test to get it working. So far I am able to get output produced including having the splittable DoFn state saved and restored for the next round of execution. I do see most of output via the `LoggingDoFn` that I added to the test but the `PAssert` is failing because it doesn't see any of the output and it is also triggering too early as not all the output has been produced. Any suggestions as to what to take a look at?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org