You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/15 08:15:19 UTC

[GitHub] [iceberg] ajantha-bhat opened a new pull request, #5284: Spark-3.3: Handle dynamic catalog configurations

ajantha-bhat opened a new pull request, #5284:
URL: https://github.com/apache/iceberg/pull/5284

   Some catalogs like Nessie supports dynamic catalog configurations (like changing the branch name).
   But catalogs obtained from spark is created always uses the original configuration (from SQL conf) instead of active session configuration. Hence, it needs an update.
   
   Nessie end to end testcases (with Nessie spark extensions) are not maintained in the Iceberg repo. Hence, not added a testcase here. Will be added in the Nessie repo.
   
   original issue from Nessie: https://github.com/projectnessie/nessie/issues/4589
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] ajantha-bhat commented on pull request #5284: [WIP] Spark-3.3: Handle dynamic catalog configurations

Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #5284:
URL: https://github.com/apache/iceberg/pull/5284#issuecomment-1205529731

   Found it. 
   Fixed by https://github.com/apache/iceberg/pull/5247 by not using SparkCatalog for the rewriteDataFIles, which was using SQLConf instead of SessionConf for building the IcebergCatalog. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] ajantha-bhat closed pull request #5284: [WIP] Spark-3.3: Handle dynamic catalog configurations

Posted by GitBox <gi...@apache.org>.
ajantha-bhat closed pull request #5284: [WIP] Spark-3.3: Handle dynamic catalog configurations
URL: https://github.com/apache/iceberg/pull/5284


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] ajantha-bhat commented on pull request #5284: [WIP] Spark-3.3: Handle dynamic catalog configurations

Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #5284:
URL: https://github.com/apache/iceberg/pull/5284#issuecomment-1205241012

   Somehow this problem disappeared.
   Iceberg 0.14.0 spark3.2 and spark3.3 doesn't have this issue anymore.
   
   Only spark 3.1 has an issue.
   
   I still need to figure out which PR fixed this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] ajantha-bhat commented on pull request #5284: [WIP] Spark-3.3: Handle dynamic catalog configurations

Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #5284:
URL: https://github.com/apache/iceberg/pull/5284#issuecomment-1186776179

   > I think this is incorrect.
   This attempts to allow a catalog to use the active session instead of the one that created it, which is not correct.
   
   @rdblue: I see that other catalogs doesn't support the Dynamic catalog properties. So, the base problem is Nessie supporting configuring reference name dynamically as a spark catalog conf?
   
   Any suggestion to handle this?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org