You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/05/05 19:05:02 UTC

[GitHub] [iceberg] singhpk234 opened a new pull request, #4707: Doc : Document time travel via SPARK SQL

singhpk234 opened a new pull request, #4707:
URL: https://github.com/apache/iceberg/pull/4707

   At present `AS OF` grammar is not supported in SPARK-SQL (was added in Spark 3.3 via [SPARK-37219](https://issues.apache.org/jira/browse/SPARK-37219)).
   
   On digging more in the code found that there was a work-around adopted by the community until this grammar is supported in SPARK itself, by using `at_timestamp_`, `snapshot_id_` 
   https://github.com/apache/iceberg/blob/bf582ebf68ea05be26f38d786584d474aebe048b/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java#L545-L555
   
   
   Presently the docs stated `Time travel is not yet supported by Spark's SQL syntax.` which can be worked around with the functionalities present. This PR attempts to document that.
   
   --- 
   
   cc @rdblue @jackye1995  @RussellSpitzer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] singhpk234 closed pull request #4707: Doc : Document time travel via SPARK SQL

Posted by GitBox <gi...@apache.org>.
singhpk234 closed pull request #4707: Doc : Document time travel via SPARK SQL
URL: https://github.com/apache/iceberg/pull/4707


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] puchengy commented on pull request #4707: Doc : Document time travel via SPARK SQL

Posted by GitBox <gi...@apache.org>.
puchengy commented on PR #4707:
URL: https://github.com/apache/iceberg/pull/4707#issuecomment-1157809116

   @singhpk234 Hi I found this issue helpful as I want to use time travel in SQL as well.
   
   Can you explain https://github.com/apache/iceberg/pull/4707#issuecomment-1118969181 as I don't see the connection between your comment ("don't want user to actually use it") and the referred comments you use.
   
   Do you mean this workaround is not guaranteed to be supported and might be dropped in the future?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] puchengy commented on pull request #4707: Doc : Document time travel via SPARK SQL

Posted by GitBox <gi...@apache.org>.
puchengy commented on PR #4707:
URL: https://github.com/apache/iceberg/pull/4707#issuecomment-1157946325

   @singhpk234 got it, thanks for your reponse.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] singhpk234 commented on pull request #4707: Doc : Document time travel via SPARK SQL

Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #4707:
URL: https://github.com/apache/iceberg/pull/4707#issuecomment-1118969181

   Looks like this was intended for internal use and we don't want the users to actually use it. 
   
   refering  : https://github.com/apache/iceberg/pull/3269#issuecomment-961520437


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] singhpk234 commented on pull request #4707: Doc : Document time travel via SPARK SQL

Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #4707:
URL: https://github.com/apache/iceberg/pull/4707#issuecomment-1157841341

   @puchengy, I was referring to this subpart of https://github.com/apache/iceberg/pull/3269#issuecomment-961520437 where @rdblue suggested :
   > We don't have to document that we support time travel through table identifiers so that we can remove it later and replace it with the AS OF syntax
   
   Hence I closed this PR considering it not being documented is intentional. 
   
   > Do you mean this workaround is not guaranteed to be supported and might be dropped in the future?
   
   As per my understanding of the comment yes, hence not documenting this could avoid users to make dependency on it.
   
   Now `AS OF` Syntax is present since [3.3](https://issues.apache.org/jira/browse/SPARK-37219), Though I am not sure of the plans of spark community to backport it back to lower spark versions like 3.0, 3.1, 3.2. Also not sure if it will be dropped from iceberg once they are supported in lower version or in 3.3. Maybe the  authors / reviewers of the pr's have deeper insights on this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org