[jira] [Created] (SPARK-16253) make spark sql compatible with hive sql that using python script transform like using 'xxx.py'

["zenglinxi (JIRA)" <ji...@apache.org>]

zenglinxi created SPARK-16253:
---------------------------------

             Summary: make spark sql compatible with hive sql that using python script transform like using 'xxx.py'
                 Key: SPARK-16253
                 URL: https://issues.apache.org/jira/browse/SPARK-16253
             Project: Spark
          Issue Type: Task
          Components: SQL
    Affects Versions: 1.6.2
            Reporter: zenglinxi


Some hive sql like:
{quote}
add file /tmp/spark_sql_test/test.py;
select transform(cityname) using 'test.py' as (new_cityname) from test.spark2_orc where dt='20160622' limit 5 ;
{quote}
can't be executed by spark sql directly, since it will return error like:
{quote}
16/06/26 11:01:28 INFO codegen.GenerateUnsafeProjection: Code generated in 19.054534 ms
16/06/26 11:01:28 ERROR execution.ScriptTransformationWriterThread: /bin/bash: test.py: command not found
{quote}
and the sql works fine in hive with MR.
Lots of ETL can't be moved from hive to spark sql because of this problem. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org