[GitHub] [spark] HeartSaVioR commented on issue #24958: [SPARK-28153][PYTHON] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

[GitBox <gi...@apache.org>]

HeartSaVioR commented on issue #24958: [SPARK-28153][PYTHON] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/24958#issuecomment-514026113
 
 
   I'd tend to agree with @hvanhovell - as there's similar case in Kafka as well and we postponed to deal with it. https://github.com/apache/spark/pull/24738#issuecomment-502764757
   
   IMHO it would be ideal to let each row have its own data for non-exposed columns (metadata, hidden columns, whatever) along with origin columns. For the case of Kafka, `offset` would be different for every rows and we can't deal with such side channel.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org