[GitHub] [spark] sc-abhisheksoni commented on issue #20393: [SPARK-23207][SQL] Shuffle+Repartition on a DataFrame could lead to incorrect answers

[GitBox <gi...@apache.org>]

sc-abhisheksoni commented on issue #20393: [SPARK-23207][SQL] Shuffle+Repartition on a DataFrame could lead to incorrect answers
URL: https://github.com/apache/spark/pull/20393#issuecomment-518447174
 
 
   I encountered this bug in my code on databricks:
   20-50 Node cluster was using - Spark 2.4.3, Scala 2.11
   
   I had a Dataframe with records for Unique IDs. Once  the data was ready to write to BLOB storage, I repartitioned the data to 2 partitions and wrote to storage. 
   
   On reading the records back from storage the number of total records remained the same but the number of Unique Ids reduced. On looking at the data the repartitioning introduced duplicate records in the data that was written to BLOB.
   As descriibed above the problem was non deterministic, and sometimes we got correct number of unique records while other times some data was duplicated.
   
   Once I removed Repartitioning I have not encountered this issue again.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org