[GitHub] spark pull request: [SPARK-2096][SQL] support dot notation on arbi...

[marmbrus <gi...@git.apache.org>]

Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/2405#issuecomment-57566803
  
    Okay here are some thoughts and questions:
     - I don't think it really matters that we can't handle `f1.f11 > f2.f22` because we already don't know what do to if a user does `[1,2] > [0,3]` even without this new syntax.
     - Am I correct in saying that hive doesn't support this syntax at all and that we are inventing new functionality?  I'm not strictly opposed to this, but we should be careful as once we support something we can't get rid of it later.
     - I'm not convinced that we need to handle arbitrary array nesting here.  The case of getting all of one field from an array (which i guess makes this SQL short hand for `array.map(_.fieldName)`) seems reasonable, but is there a use case for the arbitrary nesting version?
     - This ends up complicating `GetField` quite a bit.  What about creating a new expression type `ArrayGetField` and adding something to the analyzer that switches expression types when an array is detected.  The idea here is to keep each expression simple so we can code-gen on a case by case basis.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org