You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/01/02 18:48:00 UTC
[jira] [Commented] (SPARK-22940) Test suite
HiveExternalCatalogVersionsSuite fails on platforms that don't have wget
installed
[ https://issues.apache.org/jira/browse/SPARK-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308491#comment-16308491 ]
Sean Owen commented on SPARK-22940:
-----------------------------------
Agreed. Many other scripts require wget, though at least one will use either. The easiest option is just to document that you need wget, which is simple enough. I forget whether I have it via XCode tools or brew, but, brew install wget should always be sufficient.
Using a library is just fine too and probably more desirable within the test code.
Feel free to open a PR for any of the options, I'd say.
> Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-22940
> URL: https://issues.apache.org/jira/browse/SPARK-22940
> Project: Spark
> Issue Type: Bug
> Components: Tests
> Affects Versions: 2.2.1
> Environment: MacOS Sierra 10.12.6
> Reporter: Bruce Robbins
> Priority: Minor
>
> On platforms that don't have wget installed (e.g., Mac OS X), test suite org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite throws an exception and aborts:
> java.io.IOException: Cannot run program "wget": error=2, No such file or directory
> HiveExternalCatalogVersionsSuite uses wget to download older versions of Spark for compatibility testing. First it uses wget to find a suitable mirror, and then it uses wget to download a tar file from the mirror.
> There are several ways to fix this (in reverse order of difficulty of implementation)
> 1. Require Mac OS X users to install wget if they wish to run unit tests (or at the very least if they wish to run HiveExternalCatalogVersionsSuite). Also, update documentation to make this requirement explicit.
> 2. Fall back on curl when wget is not available.
> 3. Use an HTTP library to query for a suitable mirror and download the tar file.
> Number 2 is easy to implement, and I did so to get the unit test to run. But it relies on another external program if wget is not installed.
> Number 3 is probably slightly more complex to implement and requires more corner-case checking (e.g, redirects, etc.).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org