[jira] [Resolved] (SPARK-16253) make spark sql compatible with hive sql that using python script transform like using 'xxx.py'

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-16253.
-------------------------------
    Resolution: Not A Problem

> make spark sql compatible with hive sql that using python script transform like using 'xxx.py'
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16253
>                 URL: https://issues.apache.org/jira/browse/SPARK-16253
>             Project: Spark
>          Issue Type: Task
>          Components: SQL
>    Affects Versions: 1.6.2
>            Reporter: zenglinxi
>
> Some hive sql like:
> {quote}
> add file /tmp/spark_sql_test/test.py;
> select transform(cityname) using 'test.py' as (new_cityname) from test.spark2_orc where dt='20160622' limit 5 ;
> {quote}
> can't be executed by spark sql directly, since it will return error like:
> {quote}
> 16/06/26 11:01:28 INFO codegen.GenerateUnsafeProjection: Code generated in 19.054534 ms
> 16/06/26 11:01:28 ERROR execution.ScriptTransformationWriterThread: /bin/bash: test.py: command not found
> {quote}
> and the sql works fine in hive with MR.
> Lots of ETL can't be moved from hive to spark sql because of this problem. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org