You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2017/11/07 10:46:00 UTC
[jira] [Resolved] (SPARK-22376) run-tests.py fails at exec-sbt if
run with Python 3
[ https://issues.apache.org/jira/browse/SPARK-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-22376.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
Issue resolved by pull request 19665
[https://github.com/apache/spark/pull/19665]
> run-tests.py fails at exec-sbt if run with Python 3
> ---------------------------------------------------
>
> Key: SPARK-22376
> URL: https://issues.apache.org/jira/browse/SPARK-22376
> Project: Spark
> Issue Type: Bug
> Components: Tests
> Affects Versions: 2.3.0
> Environment: OSX 10.12.6 Python 3.6.0 Anaconda 4.4
> Reporter: Joel Croteau
> Priority: Minor
> Fix For: 2.3.0
>
>
> Running ./dev/run-tests with python3 as the default gives this error at the Building Spark stage:
> {noformat}
> ========================================================================
> Building Spark
> ========================================================================
> [info] Building Spark (w/Hive 1.2.1) using SBT with these arguments: -Phadoop-2.6 -Phive-thriftserver -Phive -Pkafka-0-8 -Pflume -Pyarn -Pmesos -Pkinesis-asl test:package streaming-kafka-0-8-assembly/assembly streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly
> Traceback (most recent call last):
> File "./dev/run-tests.py", line 622, in <module>
> main()
> File "./dev/run-tests.py", line 593, in main
> build_apache_spark(build_tool, hadoop_version)
> File "./dev/run-tests.py", line 391, in build_apache_spark
> build_spark_sbt(hadoop_version)
> File "./dev/run-tests.py", line 344, in build_spark_sbt
> exec_sbt(profiles_and_goals)
> File "./dev/run-tests.py", line 293, in exec_sbt
> if not sbt_output_filter.match(line):
> TypeError: cannot use a string pattern on a bytes-like object
> {noformat}
> This is because in Python 3, the stdout member of a POpen object defaults to returning a byte stream, and exec_sbt tries to read it as a text screen. This can be fixed by specifying universal_newlines=True when creating the POpen object. I notice that the hashbang at the start of run-tests.py says to run with python2, so I am not sure how much of the rest of it is Python 3 compatible, but this is the first error I've run into. It ran with python 3 because run-tests runs run-tests.py using the default Python, which on my system is Python 3. Not sure whether the better solution here is to try and fix Python 3 compatibility in run-tests.py or set run-tests to use Python 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org