You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/01 10:42:17 UTC
[GitHub] [iceberg] ajantha-bhat opened a new pull request, #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
ajantha-bhat opened a new pull request, #6093:
URL: https://github.com/apache/iceberg/pull/6093
This PR is a prerequisite for removing spark/v3.0 folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#discussion_r1014892191
##########
docs/aws.md:
##########
@@ -488,7 +488,7 @@ disaster recovery, etc.
For using cross-region access points, we need to additionally set `use-arn-region-enabled` catalog property to
`true` to enable `S3FileIO` to make cross-region calls, it's not required for same / multi-region access points.
-For example, to use S3 access-point with Spark 3.0, you can start the Spark SQL shell with:
+For example, to use S3 access-point with Spark 3.3, you can start the Spark SQL shell with:
Review Comment:
What about using 3.x? Or maybe just omit the version number?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#discussion_r1013168926
##########
.github/labeler.yml:
##########
@@ -61,7 +61,6 @@ DATA:
- data/**/*
SPARK:
- spark-runtime/**/*
Review Comment:
Technically can't we get rid of most of these now? since everything is spark/**/* now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#discussion_r1013167274
##########
spark/v2.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java:
##########
@@ -283,7 +283,7 @@ private Dataset<Row> createDataset(Iterable<Record> records, Schema schema) thro
@Test
public void testNullableWithWriteOption() throws IOException {
Assume.assumeTrue(
- "Spark 3.0 rejects writing nulls to a required column", spark.version().startsWith("2"));
+ "Spark 3.x rejects writing nulls to a required column", spark.version().startsWith("2"));
Review Comment:
here and in a few other places I think it may be ok to just put in Spark 3 rather than 3.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#issuecomment-1302421124
> This looks good to me, but I'm not quite up to date with our timeline for Spark 3 removal. Is this something we are doing right now? I don't have any feelings against this just want to make sure i'm in step with community decisions.
This was already discussed (notified) in the mailing list two days ago.
https://lists.apache.org/thread/bf84ovvjh6gxst562lrw5rcl966w9cy0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
rdblue merged PR #6093:
URL: https://github.com/apache/iceberg/pull/6093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#discussion_r1013163719
##########
README.md:
##########
@@ -72,8 +72,7 @@ Iceberg table support is organized in library modules:
Iceberg also has modules for adding Iceberg support to processing engines:
-* `iceberg-spark2` is an implementation of Spark's Datasource V2 API in 2.4 for Iceberg (use iceberg-spark-runtime for a shaded version)
Review Comment:
whoops, this should have been gone a while ago :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#issuecomment-1301638298
cc: @RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#issuecomment-1299482290
cc: @aokolnychyi, @rdblue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#discussion_r1013174550
##########
spark/v2.4/spark/src/test/java/org/apache/iceberg/examples/README.md:
##########
@@ -164,7 +164,7 @@ Code examples can be found [here](SnapshotFunctionalityTest.java).
Iceberg provides support to handle schema evolution of your tables over time:
1. Add a new column
- 1. The new column is always added at the end of the table (**NOTE**: This will be fixed with Spark 3.0 which has implemented AFTER and FIRST operations).
+ 1. The new column is always added at the end of the table (**NOTE**: This will be fixed with Spark 3.x which has implemented AFTER and FIRST operations).
Review Comment:
slight rewording here "This is fixed in Spark 3 ...."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #6093: Spark-3.0: Remove/update spark-3.0 mention from Docs and Builds
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on code in PR #6093:
URL: https://github.com/apache/iceberg/pull/6093#discussion_r1013172577
##########
.github/labeler.yml:
##########
@@ -61,7 +61,6 @@ DATA:
- data/**/*
SPARK:
- spark-runtime/**/*
Review Comment:
I thought of it. This file is based on a very old folder structure (not just spark folder problems). So, I thought of doing this file clean-up in the follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org