You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by nevillelyh <gi...@git.apache.org> on 2015/06/30 05:58:11 UTC
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
GitHub user nevillelyh opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/132
ZEPPELIN-142 add SparkSqlCompleter
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nevillelyh/incubator-zeppelin neville/ZEPPELIN-142
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/132.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #132
----
commit ebe6906aa2897f05fdd86c8b46d2f1cc2dd6dc6e
Author: Neville Li <ne...@spotify.com>
Date: 2015-06-30T03:18:00Z
ZEPPELIN-142 add SparkSqlCompleter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by nevillelyh <gi...@git.apache.org>.
Github user nevillelyh commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-118936916
It might be hard and unreadable since I rely on types `StructField` and `StructType` being available in the compiler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by nevillelyh <gi...@git.apache.org>.
Github user nevillelyh commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-118432538
Yeah I'm using API introduces in Spark SQL 1.3. What's the best way to make this work? Separate src/man/java directories for each spark version + Maven profiles?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-119212035
Hmm. I'm worried about separating src directory per spark version will make things too complicated.
And in the future, Zeppelin might have ability to work with user provided spark dependency. (like spark has package for user provided hadoop). It'll be difficult to support such case when codes are separated per spark version.
What do you think? Which one do you think better?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by nevillelyh <gi...@git.apache.org>.
Github user nevillelyh commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-118028970
@nberserk you invoke it with `ctrl-.` in a `%sql` note.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by nberserk <gi...@git.apache.org>.
Github user nberserk commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-118028428
when this completion invoked? I wonder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by nevillelyh <gi...@git.apache.org>.
Github user nevillelyh commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-119670370
Yeah make sense. I'll try to refactor it and remove the pattern matching parts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-118648894
Can we go with reflection?
for example, api of getting progress in spark vary between versions. And it is handled using reflections, like https://github.com/apache/incubator-zeppelin/blob/branch-0.5/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L653
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by nevillelyh <gi...@git.apache.org>.
Github user nevillelyh commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-118026764
Looks like this failed for spark-sql < 1.3. What's the recommended way of dealing with multiple Spark API versions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-142 add SparkSqlComplete...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/132#issuecomment-118431963
Thanks for the great contribution.
According to message from CI server
```
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:8: error: object StructType is not a member of package org.apache.spark.sql.types
[INFO] import org.apache.spark.sql.types.{StructType, StructField}
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:15: error: value implicits is not a member of org.apache.spark.sql.SQLContext
[INFO] import sqlContext.implicits._
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:20: error: value tables is not a member of org.apache.spark.sql.SQLContext
[INFO] override def get(): Seq[String] = sqlContext.tables().select('tableName).collect().map(_.getString(0))
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:40: error: not found: type StructField
[INFO] def fieldNames(prefix: Seq[String], fields: Array[StructField]): Seq[String] = {
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:41: error: type mismatch;
[ERROR] found : fields.type (with underlying type Array[<error>])
[INFO] required: ?{def flatMap: ?}
[INFO] Note that implicit conversions are not applicable because they are ambiguous:
[INFO] both method booleanArrayOps in object Predef of type (xs: Array[Boolean])scala.collection.mutable.ArrayOps[Boolean]
[INFO] and method byteArrayOps in object Predef of type (xs: Array[Byte])scala.collection.mutable.ArrayOps[Byte]
[INFO] are possible conversion functions from fields.type to ?{def flatMap: ?}
[INFO] fields.flatMap { f =>
```
It looks like there're some compiling error. @nevillelyh Do you have any idea about the error?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---