You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2023/05/29 13:19:28 UTC
[GitHub] [iceberg] pan3793 opened a new pull request, #7732: Spark 3.4: IcebergSource extends SessionConfigSupport
pan3793 opened a new pull request, #7732:
URL: https://github.com/apache/iceberg/pull/7732
This PR aims to make `IcebergSource extends SessionConfigSupport` to improve the Spark DataSource v2 API coverage.
```
/**
* A mix-in interface for {@link TableProvider}. Data sources can implement this interface to
* propagate session configs with the specified key-prefix to all data source operations in this
* session.
*
* @since 3.0.0
*/
@Evolving
public interface SessionConfigSupport extends TableProvider {
/**
* Key prefix of the session configs to propagate, which is usually the data source name. Spark
* will extract all session configs that starts with `spark.datasource.$keyPrefix`, turn
* `spark.datasource.$keyPrefix.xxx -> yyy` into `xxx -> yyy`, and propagate them to all
* data source operations in this session.
*/
String keyPrefix();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on pull request #7732: Spark 3.4: IcebergSource extends SessionConfigSupport
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7732:
URL: https://github.com/apache/iceberg/pull/7732#issuecomment-1571949699
@dramaticlly thanks, I refined the test code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on a diff in pull request #7732: Spark 3.4: IcebergSource extends SessionConfigSupport
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7732:
URL: https://github.com/apache/iceberg/pull/7732#discussion_r1213066513
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java:
##########
@@ -1953,6 +1954,62 @@ public void testTableWithInt96Timestamp() throws IOException {
}
}
+ @Test
+ public void testSessionConfigSupport() throws IOException {
Review Comment:
you are right. unfortunately, my IDEA can not recognize relocated guava classes after upgrading, so I can not benefit from the IDEA check. downgraded to recover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #7732: Spark 3.4: IcebergSource extends SessionConfigSupport
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #7732:
URL: https://github.com/apache/iceberg/pull/7732#discussion_r1212280530
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java:
##########
@@ -2055,4 +2112,10 @@ private DeleteFile writeEqDeleteFile(Table table) {
throw new RuntimeException(e);
}
}
+
+ private void createBranch(Table table, String branch) {
Review Comment:
is this method used anywhere? I cannot find it
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java:
##########
@@ -1953,6 +1954,62 @@ public void testTableWithInt96Timestamp() throws IOException {
}
}
+ @Test
+ public void testSessionConfigSupport() throws IOException {
Review Comment:
do we need to throw IOException here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on a diff in pull request #7732: Spark 3.4: IcebergSource extends SessionConfigSupport
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7732:
URL: https://github.com/apache/iceberg/pull/7732#discussion_r1213064711
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java:
##########
@@ -2055,4 +2112,10 @@ private DeleteFile writeEqDeleteFile(Table table) {
throw new RuntimeException(e);
}
}
+
+ private void createBranch(Table table, String branch) {
Review Comment:
removed. it was added during draft, forgot to clean up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on pull request #7732: Spark 3.4: IcebergSource extends SessionConfigSupport
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7732:
URL: https://github.com/apache/iceberg/pull/7732#issuecomment-1578708038
Kindly ping @RussellSpitzer @aokolnychyi @rdblue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org