You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bahir.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/12/15 18:11:05 UTC
[jira] [Commented] (BAHIR-138) Fix sql-cloudant deprecation
messages
[ https://issues.apache.org/jira/browse/BAHIR-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292950#comment-16292950 ]
ASF GitHub Bot commented on BAHIR-138:
--------------------------------------
GitHub user emlaver opened a pull request:
https://github.com/apache/bahir/pull/59
[BAHIR-138] fix deprecated warnings in sql-cloudant
_What_
Fix warnings in `DefaultSource` class, and in `CloudantStreaming` and `CloudantStreamingSelector` examples.
_How_
- Imported `spark.implicits._` to convert Spark RDD to Dataset
- Replaced deprecated `json(RDD[String])` with `json(Dataset[String])`
Improved streaming examples:
- Replaced registerTempTable with preferred createOrReplaceTempView
- Replaced !isEmpty with nonEmpty
- Use an accessible 'sales' database so users can run the example without any setup
- Fixed error message when stopping tests by adding logic to streaming receiver to not store documents in Spark memory when stream has stopped
See [BAHIR-138](https://issues.apache.org/jira/browse/BAHIR-138)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/emlaver/bahir 138-fix-deprecated-warnings-sql-cloudant
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bahir/pull/59.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #59
----
commit 88e239c8ab1a24ef39ca969cd11c1834ccaa1931
Author: Esteban Laver <em...@us.ibm.com>
Date: 2017-10-02T20:18:40Z
Fix deprecation warning messages
Streaming example fixes:
- Replaced registerTempTable with preferred createOrReplaceTempView
- Replaced !isEmpty with nonEmpty
commit 19a0e15328c6b3026320b4cf6ff9a9255cc1ac9c
Author: Esteban Laver <em...@us.ibm.com>
Date: 2017-12-15T17:44:50Z
Improved streaming tests by using an accessible 'sales' database
- Added logic to streaming receiver to not store doc in Spark memory when stream has stopped
----
> Fix sql-cloudant deprecation messages
> -------------------------------------
>
> Key: BAHIR-138
> URL: https://issues.apache.org/jira/browse/BAHIR-138
> Project: Bahir
> Issue Type: Task
> Components: Spark Structured Streaming Connectors
> Affects Versions: Spark-2.2.0
> Reporter: Esteban Laver
> Assignee: Esteban Laver
> Priority: Minor
> Labels: warnings
>
> Deprecation warnings in {{DefaultSource}}:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/classes...
> [WARNING] sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:59: method json in class DataFrameReader is deprecated: Use json(Dataset[String]) instead.
> [WARNING] val df = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING] sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:115: method json in class DataFrameReader is deprecated: Use json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING] sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:121: method json in class DataFrameReader is deprecated: Use json(Dataset[String]) instead.
> [WARNING] sqlContext.read.json(aRDD)
> [WARNING] ^
> [WARNING] sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:152: method json in class DataFrameReader is deprecated: Use json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.sparkSession.read.json(globalRDD)
> [WARNING] ^
> [WARNING] four warnings found
> {code}
> Deprecation warnings in {{CloudantStreaming}} and {{CloudantStreamingSelector}} examples:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/test-classes...
> [WARNING] sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:46: method json in class DataFrameReader is deprecated: Use json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:67: method registerTempTable in class Dataset is deprecated: Use createOrReplaceTempView(viewName) instead.
> [WARNING] changesDataFrame.registerTempTable("airportcodemapping")
> [WARNING] ^
> [WARNING] sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreamingSelector.scala:50: method json in class DataFrameReader is deprecated: Use json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] three warnings found
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)