You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2021/03/21 04:10:56 UTC
[kudu-CR] [spark] revert bump to 3.1.1
Hello Grant Henke,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/17213
to review the following change.
Change subject: [spark] revert bump to 3.1.1
......................................................................
[spark] revert bump to 3.1.1
TestKuduBackup.testRandomBackupAndRestore started failing with errors
like the following:
02:04:37.919 [ERROR - Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] (Logging.scala:94) Aborting task
org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into Parquet INT96 files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.int96RebaseModeInWrite to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during writing, to get maximum interoperability. Or set spark.sql.legacy.parquet.int96RebaseModeInWrite to 'CORRECTED' to write the datetime values as it is, if you are 100% sure that the written files will only be read by Spark 3.0+ or other systems that use Proleptic Gregorian calendar.
at org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInWrite(DataSourceUtils.scala:165) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
...
The error suggests some workarounds that are worth exploring further. In
the meantime, let's just revert the version bump, since it seems like
the bump will necessitate further discussion on how best to handle the
difference in behavior in the common case.
Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
---
M java/gradle/dependencies.gradle
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/17213/1
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
[kudu-CR] [spark] revert bump to 3.1.1
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [spark] revert bump to 3.1.1
......................................................................
Patch Set 1:
I can post the upgrade with the fix tomorrow or Monday.
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 04:54:40 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
......................................................................
Patch Set 2: Verified+1
Unrelated failure: RollingRestartArgs/RollingRestartITest.TestWorkloads/2
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 05:31:28 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 04:57:47 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17213
to look at the new patch set (#2).
Change subject: [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
......................................................................
[backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
After the bump to Spark 3.1.1, TestKuduBackup.testRandomBackupAndRestore
started failing with errors like the following:
02:04:37.919 [ERROR - Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] (Logging.scala:94) Aborting task
org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into Parquet INT96 files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.int96RebaseModeInWrite to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during writing, to get maximum interoperability. Or set spark.sql.legacy.parquet.int96RebaseModeInWrite to 'CORRECTED' to write the datetime values as it is, if you are 100% sure that the written files will only be read by Spark 3.0+ or other systems that use Proleptic Gregorian calendar.
at org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInWrite(DataSourceUtils.scala:165) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
...
Per their instructions, this sets the int96RebaseModeInWrite option.
Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
1 file changed, 1 insertion(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/17213/2
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
......................................................................
[backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
After the bump to Spark 3.1.1, TestKuduBackup.testRandomBackupAndRestore
started failing with errors like the following:
02:04:37.919 [ERROR - Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] (Logging.scala:94) Aborting task
org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into Parquet INT96 files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.int96RebaseModeInWrite to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during writing, to get maximum interoperability. Or set spark.sql.legacy.parquet.int96RebaseModeInWrite to 'CORRECTED' to write the datetime values as it is, if you are 100% sure that the written files will only be read by Spark 3.0+ or other systems that use Proleptic Gregorian calendar.
at org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInWrite(DataSourceUtils.scala:165) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
...
Per their instructions, this sets the int96RebaseModeInWrite option.
Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Reviewed-on: http://gerrit.cloudera.org:8080/17213
Reviewed-by: Grant Henke <gr...@apache.org>
Tested-by: Andrew Wong <aw...@cloudera.com>
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
1 file changed, 1 insertion(+), 0 deletions(-)
Approvals:
Grant Henke: Looks good to me, approved
Andrew Wong: Verified
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [spark] revert bump to 3.1.1
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [spark] revert bump to 3.1.1
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 04:54:22 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] revert bump to 3.1.1
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [spark] revert bump to 3.1.1
......................................................................
Patch Set 1:
> Patch Set 1:
>
> I am seeing
>
> 16:38:47.205 [ERROR - ForkJoinPool-169-worker-1] (KuduRestore.scala:205) Failed to restore table missingTable
> java.lang.RuntimeException: No valid backups found for table: missingTable
> at org.apache.kudu.backup.KuduRestore$.doRestore(KuduRestore.scala:56) ~[main/:?]
> at org.apache.kudu.backup.KuduRestore$.$anonfun$run$5(KuduRestore.scala:197) ~[main/:?]
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.13.jar:?]
> at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.13.jar:?]
> at org.apache.kudu.backup.KuduRestore$.$anonfun$run$4(KuduRestore.scala:197) ~[main/:?]
> at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116) ~[scala-library-2.12.13.jar:?]
> at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113) ~[scala-library-2.12.13.jar:?]
Yeah I saw the stack traces with "missingTable" but ignored them because it seems like that's only used in some specific test cases, but the test logs I was seeing (http://dist-test.cloudera.org/job?job_id=jenkins-slave.1616171649.2007161) spit out:
Time: 179.1
There was 1 failure:
1) testRandomBackupAndRestore(org.apache.kudu.backup.TestKuduBackup)
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.apache.kudu.backup.TestKuduBackup.testRandomBackupAndRestore(TestKuduBackup.scala:278)
which doesn't use "missingTable" at all (at least AFAICT).
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 04:35:14 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] revert bump to 3.1.1
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [spark] revert bump to 3.1.1
......................................................................
Patch Set 1:
I am not sure that error is the exact issue as it was set and worked around in a previous upgrade:
https://github.com/apache/kudu/commit/b2039d515bba83fa874f6d5ee7ad2bfc56e1e267
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 04:16:47 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] revert bump to 3.1.1
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [spark] revert bump to 3.1.1
......................................................................
Patch Set 1:
I am seeing
16:38:47.205 [ERROR - ForkJoinPool-169-worker-1] (KuduRestore.scala:205) Failed to restore table missingTable
java.lang.RuntimeException: No valid backups found for table: missingTable
at org.apache.kudu.backup.KuduRestore$.doRestore(KuduRestore.scala:56) ~[main/:?]
at org.apache.kudu.backup.KuduRestore$.$anonfun$run$5(KuduRestore.scala:197) ~[main/:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.13.jar:?]
at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.13.jar:?]
at org.apache.kudu.backup.KuduRestore$.$anonfun$run$4(KuduRestore.scala:197) ~[main/:?]
at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116) ~[scala-library-2.12.13.jar:?]
at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113) ~[scala-library-2.12.13.jar:?]
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 04:19:33 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has removed a vote on this change.
Change subject: [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [spark] revert bump to 3.1.1
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/17213 )
Change subject: [spark] revert bump to 3.1.1
......................................................................
Patch Set 1:
Oh I set it now. spark.sql.legacy.parquet.int96RebaseModeInWrite is different than the other legacy one I linked. I think we want the same approach in the same places as we set `session.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "LEGACY")`
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sun, 21 Mar 2021 04:41:46 +0000
Gerrit-HasComments: No