You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Vijay (JIRA)" <ji...@apache.org> on 2017/06/22 19:25:00 UTC
[jira] [Created] (SPARK-21182) Structured streaming on Spark-shell
on windows
Vijay created SPARK-21182:
-----------------------------
Summary: Structured streaming on Spark-shell on windows
Key: SPARK-21182
URL: https://issues.apache.org/jira/browse/SPARK-21182
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.1.1
Environment: Windows 10
spark-2.1.1-bin-hadoop2.7
Reporter: Vijay
Priority: Minor
Structured streaming output operation is failing on Windows shell.
As per the error message, path is being prefixed with File separator as in Linux.
Thus, causing the IllegalArgumentException.
Following is the error message.
scala> val query = wordCounts.writeStream .outputMode("complete") .format("console") .start()
java.lang.IllegalArgumentException: Pathname {color:red}*/*{color}C:/Users/Vijay/AppData/Local/Temp/temporary-081b482c-98a4-494e-8cfb-22d966c2da01/offsets from C:/Users/Vijay/AppData/Local/Temp/temporary-081b482c-98a4-494e-8cfb-22d966c2da01/offsets is not a valid DFS filename.
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:197)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:222)
at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:280)
at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:268)
... 52 elided
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org