You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shixiong Zhu (JIRA)" <ji...@apache.org> on 2017/02/23 22:11:44 UTC
[jira] [Updated] (SPARK-19718) Fix flaky test:
org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite:
stress test for failOnDataLoss=false
[ https://issues.apache.org/jira/browse/SPARK-19718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu updated SPARK-19718:
---------------------------------
Description:
SPARK-19617 changed HDFSMetadataLog to enable interrupts when using the local file system. However, now we hit HADOOP-12074: `Shell.runCommand` converts `InterruptedException` to `new IOException(ie.toString())` before Hadoop 2.8.
Test failure: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/2504/consoleFull
{code}
[info] - stress test for failOnDataLoss=false *** FAILED *** (1 minute, 1 second)
[info] org.apache.spark.sql.streaming.StreamingQueryException: Query [id = 27d45f4f-14dc-4c74-8b52-4bbd4f2b9bec, runId = 23b8c1ea-4da9-4096-967a-692933e4b319] terminated with exception: java.lang.InterruptedException
[info] at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:304)
[info] at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:190)
[info] Cause: java.io.IOException: java.lang.InterruptedException
[info] at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
[info] at org.apache.hadoop.util.Shell.run(Shell.java:478)
[info] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766)
[info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:859)
[info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:842)
[info] at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:661)
[info] at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:300)
[info] at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1014)
[info] at org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:85)
[info] at org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.<init>(ChecksumFs.java:354)
[info] at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:394)
[info] at org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:577)
[info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:680)
[info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:676)
[info] at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
[info] at org.apache.hadoop.fs.FileContext.create(FileContext.java:676)
{code}
> Fix flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite: stress test for failOnDataLoss=false
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19718
> URL: https://issues.apache.org/jira/browse/SPARK-19718
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Shixiong Zhu
>
> SPARK-19617 changed HDFSMetadataLog to enable interrupts when using the local file system. However, now we hit HADOOP-12074: `Shell.runCommand` converts `InterruptedException` to `new IOException(ie.toString())` before Hadoop 2.8.
> Test failure: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/2504/consoleFull
> {code}
> [info] - stress test for failOnDataLoss=false *** FAILED *** (1 minute, 1 second)
> [info] org.apache.spark.sql.streaming.StreamingQueryException: Query [id = 27d45f4f-14dc-4c74-8b52-4bbd4f2b9bec, runId = 23b8c1ea-4da9-4096-967a-692933e4b319] terminated with exception: java.lang.InterruptedException
> [info] at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:304)
> [info] at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:190)
> [info] Cause: java.io.IOException: java.lang.InterruptedException
> [info] at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
> [info] at org.apache.hadoop.util.Shell.run(Shell.java:478)
> [info] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766)
> [info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:859)
> [info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:842)
> [info] at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:661)
> [info] at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:300)
> [info] at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1014)
> [info] at org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:85)
> [info] at org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.<init>(ChecksumFs.java:354)
> [info] at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:394)
> [info] at org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:577)
> [info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:680)
> [info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:676)
> [info] at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> [info] at org.apache.hadoop.fs.FileContext.create(FileContext.java:676)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org