[jira] [Created] (SPARK-30855) Issue with Binding attribute: Couldn't find _gen_alias_ when using 'explode' function.

["Benoit Roy (Jira)" <ji...@apache.org>]

Benoit Roy created SPARK-30855:
----------------------------------

             Summary: Issue with Binding attribute: Couldn't find _gen_alias_ when using 'explode' function.
                 Key: SPARK-30855
                 URL: https://issues.apache.org/jira/browse/SPARK-30855
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Benoit Roy
         Attachments: msgpack-datasource.zip

Im in the early phase of writting a custom spark datasource for messagepack.

I have a basic implementation running in most cases but have encountered this issue when I tried to implement a UnaryExpression, in much the same way 'from _json' and 'from_avro' do, that converts a binary value (holding messagepack) into spark catalyst types.

When using 'from_msgpack', which returns one InternalRow, all is well. However, if I use 'from_msgpack2' which uses 'explode' to return multiple InternalRow, any select follow up selects within the df results in a [org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree: _gen_alias_40#40] error.  [Couldn't find _gen_alias_40#40 in [result#4]]

 

I've attached the code base as an archive to help you reproduce the issue.  The test class org.apache.spark.sql.messagepack.MessagePackSparkIssue contains test cases that will help you reproduce the exception.

 

It is entirely possible, and overall likely, that this is caused by a bad implementation, or wrong expectation, on my part.  Any guidance help, feedback on the cause of this exception would be greatly appreciated.  I've scoured the traditional sources for help to resolve this but haven't found anything related this this case.

 

Cheers and Thanks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org