You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/03/15 16:40:08 UTC

[GitHub] [spark] arunmahadevan commented on issue #23749: [SPARK-26841][SQL] Kafka timestamp pushdown

arunmahadevan commented on issue #23749: [SPARK-26841][SQL] Kafka timestamp pushdown
URL: https://github.com/apache/spark/pull/23749#issuecomment-473358007
 
 
   IMO, we could adopt the more generic solution. If its just timestamp based filtering of start and end offsets maybe the timestamp offset approach proposed in #23747 looks straightforward. However if there are more cases that the filter pushdown might be able to handle we should go with that. I am assuming the filter condition can appear anywhere in the query and get pushed down to filter the rows and if so looks more generic and avoids having to add extra options to the kafka source.
   
   I am not sure providing two different options for timestamp based filtering is necessary. If we support both, the user can provide different options via the options and the filter and it gets very confusing.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org