You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "rejeb (via GitHub)" <gi...@apache.org> on 2024/08/18 14:24:49 UTC
[PR] PHOENIX-6783 Add spark-sql for spark2 and spark3 [phoenix-connectors]
rejeb opened a new pull request, #139:
URL: https://github.com/apache/phoenix-connectors/pull/139
- Add spark-sql support.
- Fix support for multitenant tables with Datasource V2.
- Adapt multitenant IT to use datasource V2.
- Replace deprecated ZOOKEEPER_URL propety for all tests.
To discuss: since datasouce v1 is deprecated since long time and there is no tests for it. I removed support for datasource V1.
TODO: update READ.ME to add examples for spark-sql and multitenant examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736629365
##########
phoenix5-spark/README.md:
##########
@@ -18,6 +18,18 @@ limitations under the License.
phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
and enables persisting DataFrames back to Phoenix.
+## Configuration properties
Review Comment:
Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736662609
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkSqlIT.scala:
##########
@@ -0,0 +1,95 @@
+package org.apache.phoenix.spark
+
+import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource
+import org.apache.spark.sql.Row
+
+import scala.collection.mutable.ListBuffer
+
+/**
+ * Note: If running directly from an IDE, these are the recommended VM parameters:
+ * -Xmx1536m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m
+ */
+class PhoenixSparkSqlIT extends AbstractPhoenixSparkIT {
+ val sqlTableName = "SQL_TABLE"
+
+ after {
+ spark.sql(s"DROP TABLE IF EXISTS $sqlTableName")
+ }
+
+ test("Can read from table using spark-sql") {
+ val expected : Array[Row] = Array(
+ Row.fromSeq(Seq(1, "test_row_1")),
+ Row.fromSeq(Seq(2, "test_row_2"))
+ )
+
+ spark.sql(s"CREATE TABLE $sqlTableName USING phoenix " +
+ s"OPTIONS ('table' 'TABLE1', '${PhoenixDataSource.JDBC_URL}' '$jdbcUrl')")
+
+ val dataFrame = spark.sql(s"SELECT * FROM $sqlTableName")
+
+ dataFrame.show()
+
+ dataFrame.collect() shouldEqual expected
+ }
+
+ test("Can read from table using spark-sql with where clause and selecting specific columns`") {
+ val expected : Array[Row] = Array(
+ Row.fromSeq(Seq("test_row_1"))
+ )
+
+ spark.sql(s"CREATE TABLE $sqlTableName USING phoenix " +
+ s"OPTIONS ('table' 'TABLE1', '${PhoenixDataSource.JDBC_URL}' '$jdbcUrl')")
+
+ val dataFrame = spark.sql(s"SELECT COL1 as LABEL FROM $sqlTableName where ID=1")
+ dataFrame.show
Review Comment:
Forgot removing this before push. Done now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1760659527
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkIT.scala:
##########
@@ -66,29 +66,6 @@ class PhoenixSparkIT extends AbstractPhoenixSparkIT {
results.toList shouldEqual checkResults
}
Review Comment:
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2355725313
> Maybe it will be better to change log level to INFO and have options/instructions to enable debug level.
We've tried that, but then we couldn't debug some test failures that only happened in the CI environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2354695368
My local runs also fail:
- Can use extraOptions to set configs for workers during writes *** FAILED *** (14 milliseconds)
20 did not equal 0 (PhoenixSparkIT.scala:407)
org.scalatest.exceptions.TestFailedException:
at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:160)
at org.scalatest.Matchers$AnyShouldWrapper.shouldEqual(Matchers.scala:6303)
at org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$17.apply$mcV$sp(PhoenixSparkIT.scala:407)
at org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$17.apply(PhoenixSparkIT.scala:378)
at org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$17.apply(PhoenixSparkIT.scala:378)
at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
at org.apache.phoenix.spark.AbstractPhoenixSparkIT.org$scalatest$BeforeAndAfter$$super$runTest(AbstractPhoenixSparkIT.scala:53)
at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:200)
at org.apache.phoenix.spark.AbstractPhoenixSparkIT.runTest(AbstractPhoenixSparkIT.scala:53)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
at org.scalatest.Suite$class.run(Suite.scala:1424)
at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
at org.apache.phoenix.spark.AbstractPhoenixSparkIT.org$scalatest$BeforeAndAfter$$super$run(AbstractPhoenixSparkIT.scala:53)
at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:241)
at org.apache.phoenix.spark.AbstractPhoenixSparkIT.org$scalatest$BeforeAndAfterAll$$super$run(AbstractPhoenixSparkIT.scala:53)
at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
at org.apache.phoenix.spark.AbstractPhoenixSparkIT.run(AbstractPhoenixSparkIT.scala:53)
at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.scalatest.Suite$class.runNestedSuites(Suite.scala:1526)
at org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:29)
at org.scalatest.Suite$class.run(Suite.scala:1421)
at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:29)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2354725764
Looks like you need to reset PhoenixTestingDataSourceWriter before using it, as now you are using it from two tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1735694996
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkDatasourceV1IT.scala:
##########
@@ -0,0 +1,807 @@
+/*
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+package org.apache.phoenix.spark
+
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil
+import org.apache.phoenix.query.QueryServices
+import org.apache.phoenix.schema.types.{PSmallintArray, PVarchar}
+import org.apache.phoenix.spark.datasource.v2.reader.PhoenixTestingInputPartitionReader
+import org.apache.phoenix.spark.datasource.v2.writer.PhoenixTestingDataSourceWriter
+import org.apache.phoenix.spark.datasource.v2.{PhoenixDataSource, PhoenixTestingDataSource, SparkSchemaUtil}
+import org.apache.phoenix.util.{ColumnInfo, SchemaUtil}
+import org.apache.spark.SparkException
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SaveMode}
+
+import java.sql.DriverManager
+import java.util.Date
+import scala.collection.mutable
+import scala.collection.mutable.ListBuffer
+
+/**
+ * Note: If running directly from an IDE, these are the recommended VM parameters:
+ * -Xmx1536m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m
+ */
+class PhoenixSparkDatasourceV1IT extends AbstractPhoenixSparkIT {
Review Comment:
It is used for other tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2301211856
Please keep the DataSource V1 code.
The removal of that is out of scope for this ticket and should be discussed beforehand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty merged PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1751341141
##########
phoenix5-spark/README.md:
##########
@@ -45,12 +57,14 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "TABLE1"))
+ .option("table", "TABLE1")
Review Comment:
Nit:
Is the is the preferred way to add options in spark ?
Are you supposed to call the method multiple times if you want to add multiple options ?
##########
phoenix5-spark/README.md:
##########
@@ -118,15 +181,16 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "INPUT_TABLE"))
+ .option("table", "INPUT_TABLE")
.load
// Save to OUTPUT_TABLE
df.write
.format("phoenix")
- .mode(SaveMode.Overwrite)
- .options(Map("table" -> "OUTPUT_TABLE"))
+ .mode(SaveMode.Append)
Review Comment:
Even if both work now, let's not change this in the example.
You may want to add a comment to indicate that either works now.
##########
phoenix5-spark/README.md:
##########
@@ -300,7 +363,7 @@ to executors as a comma-separated list against the key `phoenixConfigs` i.e (Pho
.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "Table1", "jdbcUrl" -> "jdbc:phoenix:phoenix-server:2181", "doNotMapColumnFamily" -> "true"))
+ .options(Map("table" -> "Table1", "jdbcUrl" -> "jdbc:phoenix:zkHost:zkport", "doNotMapColumnFamily" -> "true"))
Review Comment:
better remove the explicit URL mention from these examples too.
##########
phoenix5-spark/README.md:
##########
@@ -270,13 +333,13 @@ it falls back to using connection defined by hbase-site.xml.
- `"jdbcUrl"` expects a full Phoenix JDBC URL, i.e. "jdbc:phoenix" or "jdbc:phoenix:zkHost:zkport",
while `"zkUrl"` expects the ZK quorum only, i.e. "zkHost:zkPort"
- If you want to use DataSourceV1, you can use source type `"org.apache.phoenix.spark"`
-instead of `"phoenix"`, however this is deprecated as of `connectors-1.0.0`.
-The `"org.apache.phoenix.spark"` datasource does not accept the `"jdbcUrl"` parameter,
-only `"zkUrl"`
+ instead of `"phoenix"`, however this is deprecated as of `connectors-1.0.0`.
Review Comment:
This was probably my mistake, but there has never been a connectors-1.0.0 release, and there will never be.
The next release will be 6.0.0.
I'd suggest simply not mentioning the version,
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/PhoenixSparkSqlRelation.scala:
##########
@@ -0,0 +1,58 @@
+package org.apache.phoenix.spark.datasource.v2
+
+import org.apache.phoenix.spark.datasource.v2.reader.PhoenixDataSourceReader
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Row, SQLContext, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.execution.datasources.v2.DataSourceRDD
+import org.apache.spark.sql.sources.v2.DataSourceOptions
+import org.apache.spark.sql.sources.{BaseRelation, Filter, InsertableRelation, PrunedFilteredScan}
+import org.apache.spark.sql.types.StructType
+
+import scala.collection.JavaConverters._
+
+case class PhoenixSparkSqlRelation(
+ @transient sparkSession: SparkSession,
+ params: Map[String, String]
+ ) extends BaseRelation with PrunedFilteredScan with InsertableRelation {
+
+ override def schema: StructType = dataSourceReader.readSchema()
+
+ override def sqlContext: SQLContext = sparkSession.sqlContext
+
+ private def dataSourceReader: PhoenixDataSourceReader = new PhoenixDataSourceReader(dataSourceOptions)
+
+ private def dataSourceOptions = new DataSourceOptions(params.asJava)
+
+ override def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] = {
+ val requiredSchema = StructType(requiredColumns.flatMap(c => schema.fields.find(_.name == c)))
+
+ val reader: PhoenixDataSourceReader = dataSourceReader
+ reader.pushFilters(filters)
+ reader.pruneColumns(requiredSchema)
+ val rdd = new DataSourceRDD(
+ sqlContext.sparkContext,
+ reader.planInputPartitions().asScala
+ )
+ rdd.map(ir => {
+ new GenericRowWithSchema(ir.toSeq(requiredSchema).toArray, requiredSchema)
+ })
+ }
+
+
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ data
+ .write
+ .format("phoenix")
+ .option(PhoenixDataSource.TABLE, params(PhoenixDataSource.TABLE))
+ .option(PhoenixDataSource.JDBC_URL, PhoenixDataSource.getJdbcUrlFromOptions(dataSourceOptions))
Review Comment:
thanks.
##########
phoenix5-spark/README.md:
##########
@@ -249,14 +312,14 @@ public class PhoenixSparkWriteFromRDDWithSchema {
}
// Create a DataFrame from the rows and the specified schema
- df = spark.createDataFrame(rows, schema);
+ Dataset<Row> df = spark.createDataFrame(rows, schema);
df.write()
.format("phoenix")
.mode(SaveMode.Overwrite)
.option("table", "OUTPUT_TABLE")
.save();
-
- jsc.stop();
+
Review Comment:
Nit:
I can't see in this editor. Is this extra whitespace ? If yes, then please delete
##########
phoenix5-spark/README.md:
##########
@@ -18,6 +18,18 @@ limitations under the License.
phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
and enables persisting DataFrames back to Phoenix.
+## Configuration properties
+
+| Name | Default | Usage | Description |
+| table | empty | R/W | table name as `namespace.table_name` |
+| zrUrl | empty | R/W | (Optional) List of zookeeper hosts. Deprecated, use `jdbcUrl` instead. Recommended not to set, value will be taken from hbase-site.xml |
Review Comment:
typo, zkUrl.
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkIT.scala:
##########
@@ -66,29 +66,6 @@ class PhoenixSparkIT extends AbstractPhoenixSparkIT {
results.toList shouldEqual checkResults
}
Review Comment:
I suggest leaving the disabled insert test in.
Hopefully it will inspire someone to implement the feature.
##########
phoenix5-spark/README.md:
##########
@@ -337,13 +401,13 @@ df.show
### Load as an RDD, using a Zookeeper URL
```scala
+import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
import org.apache.spark.rdd.RDD
-val sparkConf = new SparkConf().set("spark.ui.showConsoleProgress", "false")
-val sc = new SparkContext("local", "phoenix-test", sparkConf)
+val sc = new SparkContext("local", "phoenix-test")
Review Comment:
Nit:
You removed the showConsoleProgress setting here, but kept it below.
TBH I don't know what that does, but shouldn't it be consistent ?
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkDatasourceV1IT.scala:
##########
@@ -0,0 +1,807 @@
+/*
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+package org.apache.phoenix.spark
+
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil
+import org.apache.phoenix.query.QueryServices
+import org.apache.phoenix.schema.types.{PSmallintArray, PVarchar}
+import org.apache.phoenix.spark.datasource.v2.reader.PhoenixTestingInputPartitionReader
+import org.apache.phoenix.spark.datasource.v2.writer.PhoenixTestingDataSourceWriter
+import org.apache.phoenix.spark.datasource.v2.{PhoenixDataSource, PhoenixTestingDataSource, SparkSchemaUtil}
+import org.apache.phoenix.util.{ColumnInfo, SchemaUtil}
+import org.apache.spark.SparkException
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SaveMode}
+
+import java.sql.DriverManager
+import java.util.Date
+import scala.collection.mutable
+import scala.collection.mutable.ListBuffer
+
+/**
+ * Note: If running directly from an IDE, these are the recommended VM parameters:
+ * -Xmx1536m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m
+ */
+class PhoenixSparkDatasourceV1IT extends AbstractPhoenixSparkIT {
Review Comment:
thx
##########
phoenix5-spark3/README.md:
##########
@@ -28,6 +28,18 @@ Apart from the shaded connector JAR, you also need to add the hbase mapredcp lib
(add the exact paths as appropiate to your system)
Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Configuration properties
Review Comment:
thanks
##########
phoenix5-spark3/README.md:
##########
@@ -28,6 +28,18 @@ Apart from the shaded connector JAR, you also need to add the hbase mapredcp lib
(add the exact paths as appropiate to your system)
Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Configuration properties
+
+| Name | Default | Usage | Description |
+| table | empty | R/W | table name as `namespace.table_name` |
+| zrUrl | empty | R/W | (Optional) List of zookeeper hosts. Deprecated, use `jdbcUrl` instead. Recommended not to set, value will be taken from hbase-site.xml |
Review Comment:
typo here too.
I will not repeat the comments for the rest of the duplicated files.
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReader.java:
##########
@@ -207,6 +208,27 @@ public Filter[] pushedFilters() {
@Override
public void pruneColumns(StructType schema) {
- this.schema = schema;
+ if (schema.fields() != null && schema.fields().length != 0)
+ this.schema = schema;
+ }
+
+ //TODO Method PhoenixRuntime.generateColumnInfo skip only salt column, add skip tenant_id column.
+ private List<ColumnInfo> generateColumnInfo(Connection conn, String tableName) throws SQLException {
Review Comment:
thanks.
##########
phoenix5-spark/README.md:
##########
@@ -249,14 +307,13 @@ public class PhoenixSparkWriteFromRDDWithSchema {
}
// Create a DataFrame from the rows and the specified schema
- df = spark.createDataFrame(rows, schema);
+ Dataset<Row> df = spark.createDataFrame(rows, schema);
df.write()
.format("phoenix")
.mode(SaveMode.Overwrite)
.option("table", "OUTPUT_TABLE")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
.save();
-
Review Comment:
thx
##########
phoenix5-spark/README.md:
##########
@@ -118,15 +181,16 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "INPUT_TABLE"))
+ .option("table", "INPUT_TABLE")
.load
// Save to OUTPUT_TABLE
df.write
.format("phoenix")
- .mode(SaveMode.Overwrite)
- .options(Map("table" -> "OUTPUT_TABLE"))
+ .mode(SaveMode.Append)
Review Comment:
Also use the other examples still use Overwrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2318295160
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 10m 46s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 8s | root in master failed. |
| -1 :x: | compile | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 7s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 28s | master passed |
| -1 :x: | javadoc | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 6s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 7s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 6s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 7s | phoenix5-spark3 in master failed. |
| -0 :warning: | patch | 0m 51s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 6s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 5s | root in the patch failed. |
| -1 :x: | compile | 0m 7s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 7s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 7s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 7s | phoenix5-spark: The patch generated 15 new + 36 unchanged - 74 fixed = 51 total (was 110) |
| -1 :x: | checkstyle | 0m 8s | phoenix5-spark3: The patch generated 18 new + 40 unchanged - 80 fixed = 58 total (was 120) |
| -1 :x: | markdownlint | 0m 3s | The patch generated 38 new + 97 unchanged - 15 fixed = 135 total (was 112) |
| -1 :x: | whitespace | 0m 0s | The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 7s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 7s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 6s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 14s | The patch generated 4 ASF License warnings. |
| | | 17m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 96374444b455 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / a1c298c |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 52 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/4/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736704055
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReader.java:
##########
@@ -207,6 +208,27 @@ public Filter[] pushedFilters() {
@Override
public void pruneColumns(StructType schema) {
- this.schema = schema;
+ if (schema.fields() != null && schema.fields().length != 0)
+ this.schema = schema;
+ }
+
+ //TODO Method PhoenixRuntime.generateColumnInfo skip only salt column, add skip tenant_id column.
+ private List<ColumnInfo> generateColumnInfo(Connection conn, String tableName) throws SQLException {
Review Comment:
I already created an issue for PhoenixRuntime.generateColumnInfo. I plan to submit a PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736644267
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/FilterExpressionCompiler.scala:
##########
@@ -15,16 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.phoenix.spark
+package org.apache.phoenix.spark.datasource.v2
Review Comment:
I moved the class back to org.apache.phoenix.spark package.
For v2 package, I think it can be renamed when classes of V1 will be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2355338969
> It's a major pain to identify any problem from the 40MB scalatest output. It would be really nice if there was a newer version that split its outputs failsafe/maven style.
Unzipped file is about 450MB and most of the logs are debug logs essentially hbase logs. IDK, if it is helfull to have these logs during tests.
Maybe it will be better to change log level to INFO and have options/instructions to enable debug level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2318441490
> When working on reviews I found out that AggregateIT and SaltedTableIT are running only pure phoenix tests and there is no tests for spark connector so I disabled them and I will open an issue to add real spark tests for these classes.
That's not what I can see.
Those tests are piped through Spark, and exercise the connector.
Instead of executing directly, they are executed via Spark.
Please check again, and if you still find that they are pure Phoenix tests, point out why the code to perform the queries via Spark does not work. (executeQuery() override )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2344012720
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 13m 46s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 8s | root in master failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 32s | master passed |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 6s | root in the patch failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark: The patch generated 15 new + 35 unchanged - 74 fixed = 50 total (was 109) |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark3: The patch generated 18 new + 39 unchanged - 80 fixed = 57 total (was 119) |
| -1 :x: | markdownlint | 0m 3s | The patch generated 40 new + 95 unchanged - 17 fixed = 135 total (was 112) |
| -1 :x: | whitespace | 0m 0s | The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 17s | The patch generated 4 ASF License warnings. |
| | | 19m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux accce451f442 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 6b6a85b |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/7/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1760657529
##########
phoenix5-spark/README.md:
##########
@@ -337,13 +401,13 @@ df.show
### Load as an RDD, using a Zookeeper URL
```scala
+import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
import org.apache.spark.rdd.RDD
-val sparkConf = new SparkConf().set("spark.ui.showConsoleProgress", "false")
-val sc = new SparkContext("local", "phoenix-test", sparkConf)
+val sc = new SparkContext("local", "phoenix-test")
Review Comment:
thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1760656215
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/FilterExpressionCompiler.scala:
##########
@@ -15,16 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.phoenix.spark
+package org.apache.phoenix.spark.datasource.v2
Review Comment:
IIRC the relevant spark packages also have "v2" in the name, so having similar names in Phoenix makes sense for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1734210681
##########
phoenix5-spark/README.md:
##########
@@ -45,7 +57,7 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "TABLE1"))
+ .options(Map("table" -> "TABLE1", "jdbcUrl" -> "jdbc:phoenix:zkHost:zkport"))
Review Comment:
Specifying an explicit jdbcURL is somewhat of an anti-pattern.
Ideally, the connection info should be taken from hbase-site.xml
##########
phoenix5-spark/README.md:
##########
@@ -18,6 +18,18 @@ limitations under the License.
phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
and enables persisting DataFrames back to Phoenix.
+## Configuration properties
Review Comment:
Nice
##########
phoenix5-spark/src/it/resources/globalSetup.sql:
##########
@@ -60,7 +60,7 @@ UPSERT INTO "small" VALUES ('key1', 'foo', 10000)
UPSERT INTO "small" VALUES ('key2', 'bar', 20000)
UPSERT INTO "small" VALUES ('key3', 'xyz', 30000)
-CREATE TABLE MULTITENANT_TEST_TABLE (TENANT_ID VARCHAR NOT NULL, ORGANIZATION_ID VARCHAR, GLOBAL_COL1 VARCHAR CONSTRAINT pk PRIMARY KEY (TENANT_ID, ORGANIZATION_ID)) MULTI_TENANT=true
+CREATE TABLE MULTITENANT_TEST_TABLE (TENANT_ID VARCHAR NOT NULL, ORGANIZATION_ID VARCHAR, GLOBAL_COL1 VARCHAR CONSTRAINT pk PRIMARY KEY (TENANT_ID, ORGANIZATION_ID)) MULTI_TENANT=true, SALT_BUCKETS = 20
Review Comment:
If we want to test salting, we should create a new table and new tests for that.
##########
phoenix5-spark/README.md:
##########
@@ -249,14 +307,13 @@ public class PhoenixSparkWriteFromRDDWithSchema {
}
// Create a DataFrame from the rows and the specified schema
- df = spark.createDataFrame(rows, schema);
+ Dataset<Row> df = spark.createDataFrame(rows, schema);
df.write()
.format("phoenix")
.mode(SaveMode.Overwrite)
.option("table", "OUTPUT_TABLE")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
.save();
-
Review Comment:
Shouldn't we close the session ?
While this is only an example, I think that it should teach proper resource management.
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/PhoenixDataSource.java:
##########
@@ -33,21 +37,27 @@
import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
import org.apache.spark.sql.types.StructType;
+import scala.collection.immutable.Map;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
/**
* Implements the DataSourceV2 api to read and write from Phoenix tables
*/
-public class PhoenixDataSource implements DataSourceV2, ReadSupport, WriteSupport, DataSourceRegister {
+public class PhoenixDataSource implements DataSourceV2, ReadSupport, WriteSupport, DataSourceRegister, RelationProvider {
private static final Logger logger = LoggerFactory.getLogger(PhoenixDataSource.class);
public static final String SKIP_NORMALIZING_IDENTIFIER = "skipNormalizingIdentifier";
@Deprecated
public static final String ZOOKEEPER_URL = "zkUrl";
public static final String JDBC_URL = "jdbcUrl";
public static final String PHOENIX_CONFIGS = "phoenixconfigs";
+ public static final String TABLE = "table";
+ public static final String DATE_AS_TIME_STAMP = "dateAsTimestamp";
+ public static final String DO_NOT_MAP_COLUMN_FAMILY = "doNotMapColumnFamily";
+ public static final String TENANT_ID = "tenantId";
Review Comment:
use PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID as value instead of repeating the string ?
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/PhoenixRDD.scala:
##########
@@ -14,14 +14,14 @@
package org.apache.phoenix.spark
import java.sql.DriverManager
-
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.HConstants
import org.apache.hadoop.io.NullWritable
import org.apache.phoenix.jdbc.PhoenixDriver
import org.apache.phoenix.mapreduce.PhoenixInputFormat
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil
import org.apache.phoenix.query.HBaseFactoryProvider
+import org.apache.phoenix.spark.datasource.v2.SparkSchemaUtil
Review Comment:
looks like an unused import.
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkDatasourceV1IT.scala:
##########
@@ -0,0 +1,807 @@
+/*
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+package org.apache.phoenix.spark
+
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil
+import org.apache.phoenix.query.QueryServices
+import org.apache.phoenix.schema.types.{PSmallintArray, PVarchar}
+import org.apache.phoenix.spark.datasource.v2.reader.PhoenixTestingInputPartitionReader
+import org.apache.phoenix.spark.datasource.v2.writer.PhoenixTestingDataSourceWriter
+import org.apache.phoenix.spark.datasource.v2.{PhoenixDataSource, PhoenixTestingDataSource, SparkSchemaUtil}
+import org.apache.phoenix.util.{ColumnInfo, SchemaUtil}
+import org.apache.spark.SparkException
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{Row, SaveMode}
+
+import java.sql.DriverManager
+import java.util.Date
+import scala.collection.mutable
+import scala.collection.mutable.ListBuffer
+
+/**
+ * Note: If running directly from an IDE, these are the recommended VM parameters:
+ * -Xmx1536m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m
+ */
+class PhoenixSparkDatasourceV1IT extends AbstractPhoenixSparkIT {
Review Comment:
IIRC AbstractPhoenixSparkIT is only used by the V1 tests.
If you rename the tests, it would make sense to also rename the parent class.
##########
phoenix5-spark/README.md:
##########
@@ -89,7 +147,7 @@ public class PhoenixSparkRead {
### Save DataFrames to Phoenix using DataSourceV2
The `save` is method on DataFrame allows passing in a data source type. You can use
-`phoenix` for DataSourceV2 and must also pass in a `table` and `zkUrl` parameter to
+`phoenix` for DataSourceV2 and must also pass in a `table` and `jdbcUrl` parameter to
Review Comment:
This is not true anymore (it was already wrong, but since you're updating the docs, let's fix this).
jdbcUrl is not mandatory.
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkDatasourceV1ITTenantSpecific.scala:
##########
@@ -105,39 +102,6 @@ class PhoenixSparkITTenantSpecific extends AbstractPhoenixSparkIT {
verifyResults
}
- ignore("Can write a DataFrame using 'DataFrame.write' to tenant-specific view - Spark2 sparse columns") {
Review Comment:
I am not sure why this is ignored, but I assume that there is a bug that this would catch.
It'd be better keep this.
##########
phoenix5-spark/README.md:
##########
@@ -141,32 +201,32 @@ public class PhoenixSparkWriteFromInputTable {
public static void main() throws Exception {
SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("phoenix-test")
- .set("spark.hadoopRDD.ignoreEmptySplits", "false");
- JavaSparkContext jsc = new JavaSparkContext(sparkConf);
- SQLContext sqlContext = new SQLContext(jsc);
+ .set("spark.hadoopRDD.ignoreEmptySplits", "false");
+ SparkSessinon spark = SparkSession.builder().config(sparkConf).getOrCreate();
// Load INPUT_TABLE
- Dataset<Row> df = sqlContext
+ Dataset<Row> df = spark
.read()
.format("phoenix")
.option("table", "INPUT_TABLE")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
.load();
// Save to OUTPUT_TABLE
df.write()
.format("phoenix")
.mode(SaveMode.Overwrite)
.option("table", "OUTPUT_TABLE")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
.save();
- jsc.stop();
}
}
```
### Save from an external RDD with a schema to a Phoenix table
Just like the previous example, you can pass in the data source type as `phoenix` and specify the `table` and
-`zkUrl` parameters indicating which table and server to persist the DataFrame to.
+`jdbcUrl` parameters indicating which table and server to persist the DataFrame to.
Review Comment:
same, jdbcUrl is optional.
##########
phoenix5-spark/README.md:
##########
@@ -65,21 +77,67 @@ public class PhoenixSparkRead {
public static void main() throws Exception {
SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("phoenix-test")
.set("spark.hadoopRDD.ignoreEmptySplits", "false");
- JavaSparkContext jsc = new JavaSparkContext(sparkConf);
- SQLContext sqlContext = new SQLContext(jsc);
+ SparkSessinon spark = SparkSession.builder().config(sparkConf).getOrCreate();
// Load data from TABLE1
- Dataset<Row> df = sqlContext
+ Dataset<Row> df = spark
.read()
.format("phoenix")
.option("table", "TABLE1")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
Review Comment:
see above, this applies to all uses with explicit jdbcUrl
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReader.java:
##########
@@ -207,6 +208,27 @@ public Filter[] pushedFilters() {
@Override
public void pruneColumns(StructType schema) {
- this.schema = schema;
+ if (schema.fields() != null && schema.fields().length != 0)
+ this.schema = schema;
+ }
+
+ //TODO Method PhoenixRuntime.generateColumnInfo skip only salt column, add skip tenant_id column.
+ private List<ColumnInfo> generateColumnInfo(Connection conn, String tableName) throws SQLException {
Review Comment:
You may want to consider exposing this functionality in PhoenixRuntime, or the Phoenix MR classes, if it is not yet exposed.
This is for later.
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkSqlIT.scala:
##########
@@ -0,0 +1,95 @@
+package org.apache.phoenix.spark
+
+import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource
+import org.apache.spark.sql.Row
+
+import scala.collection.mutable.ListBuffer
+
+/**
+ * Note: If running directly from an IDE, these are the recommended VM parameters:
+ * -Xmx1536m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m
+ */
+class PhoenixSparkSqlIT extends AbstractPhoenixSparkIT {
+ val sqlTableName = "SQL_TABLE"
+
+ after {
+ spark.sql(s"DROP TABLE IF EXISTS $sqlTableName")
+ }
+
+ test("Can read from table using spark-sql") {
+ val expected : Array[Row] = Array(
+ Row.fromSeq(Seq(1, "test_row_1")),
+ Row.fromSeq(Seq(2, "test_row_2"))
+ )
+
+ spark.sql(s"CREATE TABLE $sqlTableName USING phoenix " +
+ s"OPTIONS ('table' 'TABLE1', '${PhoenixDataSource.JDBC_URL}' '$jdbcUrl')")
+
+ val dataFrame = spark.sql(s"SELECT * FROM $sqlTableName")
+
+ dataFrame.show()
+
+ dataFrame.collect() shouldEqual expected
+ }
+
+ test("Can read from table using spark-sql with where clause and selecting specific columns`") {
+ val expected : Array[Row] = Array(
+ Row.fromSeq(Seq("test_row_1"))
+ )
+
+ spark.sql(s"CREATE TABLE $sqlTableName USING phoenix " +
+ s"OPTIONS ('table' 'TABLE1', '${PhoenixDataSource.JDBC_URL}' '$jdbcUrl')")
+
+ val dataFrame = spark.sql(s"SELECT COL1 as LABEL FROM $sqlTableName where ID=1")
+ dataFrame.show
Review Comment:
While these may be useful for debugging, In general we shouldn't print to the stdout in tests.
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/PhoenixRelation.scala:
##########
@@ -18,8 +18,9 @@
package org.apache.phoenix.spark
import org.apache.hadoop.conf.Configuration
+import org.apache.phoenix.spark.datasource.v2.FilterExpressionCompiler
Review Comment:
looks like an unused import
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/PhoenixDataSource.java:
##########
@@ -56,7 +66,7 @@ public DataSourceReader createReader(DataSourceOptions options) {
@Override
public Optional<DataSourceWriter> createWriter(String writeUUID, StructType schema,
- SaveMode mode, DataSourceOptions options) {
+ SaveMode mode, DataSourceOptions options) {
Review Comment:
We don't use this indent style.
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/FilterExpressionCompiler.scala:
##########
@@ -15,16 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.phoenix.spark
+package org.apache.phoenix.spark.datasource.v2
Review Comment:
Generally, do we need the new v2 package in the Spark2 module ?
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/FilterExpressionCompiler.scala:
##########
@@ -15,16 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.phoenix.spark
+package org.apache.phoenix.spark.datasource.v2
Review Comment:
If it is still used in the legacy code, then it may be better not to move.
It is only used by the v2 connectore, then it's fine to move.
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/PhoenixSparkSqlRelation.scala:
##########
@@ -0,0 +1,58 @@
+package org.apache.phoenix.spark.datasource.v2
+
+import org.apache.phoenix.spark.datasource.v2.reader.PhoenixDataSourceReader
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Row, SQLContext, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.execution.datasources.v2.DataSourceRDD
+import org.apache.spark.sql.sources.v2.DataSourceOptions
+import org.apache.spark.sql.sources.{BaseRelation, Filter, InsertableRelation, PrunedFilteredScan}
+import org.apache.spark.sql.types.StructType
+
+import scala.collection.JavaConverters._
+
+case class PhoenixSparkSqlRelation(
+ @transient sparkSession: SparkSession,
+ params: Map[String, String]
+ ) extends BaseRelation with PrunedFilteredScan with InsertableRelation {
+
+ override def schema: StructType = dataSourceReader.readSchema()
+
+ override def sqlContext: SQLContext = sparkSession.sqlContext
+
+ private def dataSourceReader: PhoenixDataSourceReader = new PhoenixDataSourceReader(dataSourceOptions)
+
+ private def dataSourceOptions = new DataSourceOptions(params.asJava)
+
+ override def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] = {
+ val requiredSchema = StructType(requiredColumns.flatMap(c => schema.fields.find(_.name == c)))
+
+ val reader: PhoenixDataSourceReader = dataSourceReader
+ reader.pushFilters(filters)
+ reader.pruneColumns(requiredSchema)
+ val rdd = new DataSourceRDD(
+ sqlContext.sparkContext,
+ reader.planInputPartitions().asScala
+ )
+ rdd.map(ir => {
+ new GenericRowWithSchema(ir.toSeq(requiredSchema).toArray, requiredSchema)
+ })
+ }
+
+
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ data
+ .write
+ .format("phoenix")
+ .option(PhoenixDataSource.TABLE, params(PhoenixDataSource.TABLE))
+ .option(PhoenixDataSource.JDBC_URL, PhoenixDataSource.getJdbcUrlFromOptions(dataSourceOptions))
Review Comment:
does this handle the case when jdbcUrl is not specified ?
##########
phoenix5-spark3/README.md:
##########
@@ -28,6 +28,18 @@ Apart from the shaded connector JAR, you also need to add the hbase mapredcp lib
(add the exact paths as appropiate to your system)
Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Configuration properties
Review Comment:
Same comments apply as to the v2 readme,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736647542
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/PhoenixSparkSqlRelation.scala:
##########
@@ -0,0 +1,58 @@
+package org.apache.phoenix.spark.datasource.v2
+
+import org.apache.phoenix.spark.datasource.v2.reader.PhoenixDataSourceReader
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Row, SQLContext, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.execution.datasources.v2.DataSourceRDD
+import org.apache.spark.sql.sources.v2.DataSourceOptions
+import org.apache.spark.sql.sources.{BaseRelation, Filter, InsertableRelation, PrunedFilteredScan}
+import org.apache.spark.sql.types.StructType
+
+import scala.collection.JavaConverters._
+
+case class PhoenixSparkSqlRelation(
+ @transient sparkSession: SparkSession,
+ params: Map[String, String]
+ ) extends BaseRelation with PrunedFilteredScan with InsertableRelation {
+
+ override def schema: StructType = dataSourceReader.readSchema()
+
+ override def sqlContext: SQLContext = sparkSession.sqlContext
+
+ private def dataSourceReader: PhoenixDataSourceReader = new PhoenixDataSourceReader(dataSourceOptions)
+
+ private def dataSourceOptions = new DataSourceOptions(params.asJava)
+
+ override def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] = {
+ val requiredSchema = StructType(requiredColumns.flatMap(c => schema.fields.find(_.name == c)))
+
+ val reader: PhoenixDataSourceReader = dataSourceReader
+ reader.pushFilters(filters)
+ reader.pruneColumns(requiredSchema)
+ val rdd = new DataSourceRDD(
+ sqlContext.sparkContext,
+ reader.planInputPartitions().asScala
+ )
+ rdd.map(ir => {
+ new GenericRowWithSchema(ir.toSeq(requiredSchema).toArray, requiredSchema)
+ })
+ }
+
+
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ data
+ .write
+ .format("phoenix")
+ .option(PhoenixDataSource.TABLE, params(PhoenixDataSource.TABLE))
+ .option(PhoenixDataSource.JDBC_URL, PhoenixDataSource.getJdbcUrlFromOptions(dataSourceOptions))
Review Comment:
The PhoenixDataSource.getJdbcUrlFromOptions method handle passing jdbcUrl, zkUrl and also when none is specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736644267
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/FilterExpressionCompiler.scala:
##########
@@ -15,16 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.phoenix.spark
+package org.apache.phoenix.spark.datasource.v2
Review Comment:
I moved the class back to org.apache.phoenix.spark package.
For v2 package, I think it can be renamed when classes of V1 are removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1754928590
##########
phoenix5-spark/README.md:
##########
@@ -249,14 +312,14 @@ public class PhoenixSparkWriteFromRDDWithSchema {
}
// Create a DataFrame from the rows and the specified schema
- df = spark.createDataFrame(rows, schema);
+ Dataset<Row> df = spark.createDataFrame(rows, schema);
df.write()
.format("phoenix")
.mode(SaveMode.Overwrite)
.option("table", "OUTPUT_TABLE")
.save();
-
- jsc.stop();
+
Review Comment:
Fixed
##########
phoenix5-spark/README.md:
##########
@@ -18,6 +18,18 @@ limitations under the License.
phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
and enables persisting DataFrames back to Phoenix.
+## Configuration properties
+
+| Name | Default | Usage | Description |
+| table | empty | R/W | table name as `namespace.table_name` |
+| zrUrl | empty | R/W | (Optional) List of zookeeper hosts. Deprecated, use `jdbcUrl` instead. Recommended not to set, value will be taken from hbase-site.xml |
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2357575475
Updated the commit message to match the PR description on commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2355776526
Please also rename the JIRA to match the commit message (which is more descriptive).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1754929684
##########
phoenix5-spark/README.md:
##########
@@ -270,13 +333,13 @@ it falls back to using connection defined by hbase-site.xml.
- `"jdbcUrl"` expects a full Phoenix JDBC URL, i.e. "jdbc:phoenix" or "jdbc:phoenix:zkHost:zkport",
while `"zkUrl"` expects the ZK quorum only, i.e. "zkHost:zkPort"
- If you want to use DataSourceV1, you can use source type `"org.apache.phoenix.spark"`
-instead of `"phoenix"`, however this is deprecated as of `connectors-1.0.0`.
-The `"org.apache.phoenix.spark"` datasource does not accept the `"jdbcUrl"` parameter,
-only `"zkUrl"`
+ instead of `"phoenix"`, however this is deprecated as of `connectors-1.0.0`.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1755048984
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkIT.scala:
##########
@@ -66,29 +66,6 @@ class PhoenixSparkIT extends AbstractPhoenixSparkIT {
results.toList shouldEqual checkResults
}
Review Comment:
I moved the test to the class PhoenixSparkSqlIT and changed the first step for loading and creating the view.
It seems it won't work when using spark.read and createOrReplaceTempView but I copied the test just in case someone else find out a solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736648079
##########
phoenix5-spark3/README.md:
##########
@@ -28,6 +28,18 @@ Apart from the shaded connector JAR, you also need to add the hbase mapredcp lib
(add the exact paths as appropiate to your system)
Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Configuration properties
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2318551487
I haven't paid attention to the executeQuery method. You're rigth, the tests are executed throw spark. I rolled back my modifications and done the same for OrderByIT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2354513391
I have re-started the test to see if the first failure was a transient.
You can still download the full logs for the failed attempt from GH (I hopethe next run won't age it out)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2355862732
> Please also rename the JIRA to match the commit message (which is more descriptive).
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2355323138
Hi @stoty,
I should've run all tests at once instead of running them individually before submitting the PR.
I found out that the failed tests are just duplication because they use `.format(PhoenixTestingDataSource.TEST_SOURCE)` and not V1 and V2 format. However I added reset of the TOTAL_BATCHES_COMMITTED_COUNT before each test.
I went on a tricky error tests with tables `DATE_TEST` and `TIME_TEST` because they use currentdatetime and check that date is less than 24 hours difference. I changed these tests to use a fixed date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2355399157
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 14m 20s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 9s | root in master failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 33s | master passed |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 7s | root in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark: The patch generated 15 new + 35 unchanged - 74 fixed = 50 total (was 109) |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark3: The patch generated 18 new + 39 unchanged - 80 fixed = 57 total (was 119) |
| -1 :x: | markdownlint | 0m 3s | The patch generated 40 new + 95 unchanged - 17 fixed = 135 total (was 112) |
| -1 :x: | whitespace | 0m 0s | The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 17s | The patch generated 4 ASF License warnings. |
| | | 25m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 928ffb518800 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / cd2fcf0 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/8/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1754930272
##########
phoenix5-spark/README.md:
##########
@@ -300,7 +363,7 @@ to executors as a comma-separated list against the key `phoenixConfigs` i.e (Pho
.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "Table1", "jdbcUrl" -> "jdbc:phoenix:phoenix-server:2181", "doNotMapColumnFamily" -> "true"))
+ .options(Map("table" -> "Table1", "jdbcUrl" -> "jdbc:phoenix:zkHost:zkport", "doNotMapColumnFamily" -> "true"))
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736655283
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/PhoenixDataSource.java:
##########
@@ -56,7 +66,7 @@ public DataSourceReader createReader(DataSourceOptions options) {
@Override
public Optional<DataSourceWriter> createWriter(String writeUUID, StructType schema,
- SaveMode mode, DataSourceOptions options) {
+ SaveMode mode, DataSourceOptions options) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736699009
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/PhoenixDataSource.java:
##########
@@ -33,21 +37,27 @@
import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
import org.apache.spark.sql.types.StructType;
+import scala.collection.immutable.Map;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
/**
* Implements the DataSourceV2 api to read and write from Phoenix tables
*/
-public class PhoenixDataSource implements DataSourceV2, ReadSupport, WriteSupport, DataSourceRegister {
+public class PhoenixDataSource implements DataSourceV2, ReadSupport, WriteSupport, DataSourceRegister, RelationProvider {
private static final Logger logger = LoggerFactory.getLogger(PhoenixDataSource.class);
public static final String SKIP_NORMALIZING_IDENTIFIER = "skipNormalizingIdentifier";
@Deprecated
public static final String ZOOKEEPER_URL = "zkUrl";
public static final String JDBC_URL = "jdbcUrl";
public static final String PHOENIX_CONFIGS = "phoenixconfigs";
+ public static final String TABLE = "table";
+ public static final String DATE_AS_TIME_STAMP = "dateAsTimestamp";
+ public static final String DO_NOT_MAP_COLUMN_FAMILY = "doNotMapColumnFamily";
+ public static final String TENANT_ID = "tenantId";
Review Comment:
I think PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID have another usage and can't be used here.
The TENANT_ID property is only used at spark options level. For setting the value for phoenix connection property I use PhoenixRuntime.TENANT_ID_ATTRIB.
I prefer keep them separate and duplicate value because PhoenixRuntime.TENANT_ID_ATTRIB come from a phoenix dependency so if its value changes it will have an effect on spark connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736704055
##########
phoenix5-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReader.java:
##########
@@ -207,6 +208,27 @@ public Filter[] pushedFilters() {
@Override
public void pruneColumns(StructType schema) {
- this.schema = schema;
+ if (schema.fields() != null && schema.fields().length != 0)
+ this.schema = schema;
+ }
+
+ //TODO Method PhoenixRuntime.generateColumnInfo skip only salt column, add skip tenant_id column.
+ private List<ColumnInfo> generateColumnInfo(Connection conn, String tableName) throws SQLException {
Review Comment:
I already created an issue for PhoenixRuntime.generateColumnInfo. I plan to submit a Phoenix PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2302553995
Rollback datasourceV1 and add tests.
I opened this discussion [PHOENIX-7392](https://issues.apache.org/jira/browse/PHOENIX-7392) about removing datasource V1 and building connector with different scala versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1754913466
##########
phoenix5-spark/README.md:
##########
@@ -337,13 +401,13 @@ df.show
### Load as an RDD, using a Zookeeper URL
```scala
+import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
import org.apache.spark.rdd.RDD
-val sparkConf = new SparkConf().set("spark.ui.showConsoleProgress", "false")
-val sc = new SparkContext("local", "phoenix-test", sparkConf)
+val sc = new SparkContext("local", "phoenix-test")
Review Comment:
From spark doc, this flag is to enable/desable some logs in console.
Since this param has no effect on connector behavior I removed it in the other examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1755053409
##########
phoenix5-spark3/README.md:
##########
@@ -28,6 +28,18 @@ Apart from the shaded connector JAR, you also need to add the hbase mapredcp lib
(add the exact paths as appropiate to your system)
Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Configuration properties
+
+| Name | Default | Usage | Description |
+| table | empty | R/W | table name as `namespace.table_name` |
+| zrUrl | empty | R/W | (Optional) List of zookeeper hosts. Deprecated, use `jdbcUrl` instead. Recommended not to set, value will be taken from hbase-site.xml |
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1760657046
##########
phoenix5-spark/README.md:
##########
@@ -118,15 +181,16 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "INPUT_TABLE"))
+ .option("table", "INPUT_TABLE")
.load
// Save to OUTPUT_TABLE
df.write
.format("phoenix")
- .mode(SaveMode.Overwrite)
- .options(Map("table" -> "OUTPUT_TABLE"))
+ .mode(SaveMode.Append)
Review Comment:
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736626191
##########
phoenix5-spark/src/it/java/org/apache/phoenix/spark/OrderByIT.java:
##########
@@ -46,15 +46,9 @@
import org.apache.spark.sql.sources.v2.DataSourceOptions;
import org.junit.Test;
import org.junit.experimental.categories.Category;
-import org.junit.Ignore;
-
-
-
-import scala.Option;
-import scala.collection.JavaConverters;
@Category(ParallelStatsDisabledTest.class)
-public class OrderByIT extends BaseOrderByIT {
Review Comment:
Bad extends, BaseOrderByIT runs Phoenix based IT tests. These tests are out of context here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2318371743
@stoty, than you for the review. I made the changes as you suggest and added comments.
When working on reviews I found out that AggregateIT and SaltedTableIT are running only pure phoenix tests and there is no tests for spark connector so I disabled them and I will open an issue to add real spark tests for these classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736632108
##########
phoenix5-spark/README.md:
##########
@@ -45,7 +57,7 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "TABLE1"))
+ .options(Map("table" -> "TABLE1", "jdbcUrl" -> "jdbc:phoenix:zkHost:zkport"))
Review Comment:
I removed it from all example and added the comment in configuration properties section to explain it.
##########
phoenix5-spark/README.md:
##########
@@ -65,21 +77,67 @@ public class PhoenixSparkRead {
public static void main() throws Exception {
SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("phoenix-test")
.set("spark.hadoopRDD.ignoreEmptySplits", "false");
- JavaSparkContext jsc = new JavaSparkContext(sparkConf);
- SQLContext sqlContext = new SQLContext(jsc);
+ SparkSessinon spark = SparkSession.builder().config(sparkConf).getOrCreate();
// Load data from TABLE1
- Dataset<Row> df = sqlContext
+ Dataset<Row> df = spark
.read()
.format("phoenix")
.option("table", "TABLE1")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736647542
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/datasource/v2/PhoenixSparkSqlRelation.scala:
##########
@@ -0,0 +1,58 @@
+package org.apache.phoenix.spark.datasource.v2
+
+import org.apache.phoenix.spark.datasource.v2.reader.PhoenixDataSourceReader
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Row, SQLContext, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.execution.datasources.v2.DataSourceRDD
+import org.apache.spark.sql.sources.v2.DataSourceOptions
+import org.apache.spark.sql.sources.{BaseRelation, Filter, InsertableRelation, PrunedFilteredScan}
+import org.apache.spark.sql.types.StructType
+
+import scala.collection.JavaConverters._
+
+case class PhoenixSparkSqlRelation(
+ @transient sparkSession: SparkSession,
+ params: Map[String, String]
+ ) extends BaseRelation with PrunedFilteredScan with InsertableRelation {
+
+ override def schema: StructType = dataSourceReader.readSchema()
+
+ override def sqlContext: SQLContext = sparkSession.sqlContext
+
+ private def dataSourceReader: PhoenixDataSourceReader = new PhoenixDataSourceReader(dataSourceOptions)
+
+ private def dataSourceOptions = new DataSourceOptions(params.asJava)
+
+ override def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] = {
+ val requiredSchema = StructType(requiredColumns.flatMap(c => schema.fields.find(_.name == c)))
+
+ val reader: PhoenixDataSourceReader = dataSourceReader
+ reader.pushFilters(filters)
+ reader.pruneColumns(requiredSchema)
+ val rdd = new DataSourceRDD(
+ sqlContext.sparkContext,
+ reader.planInputPartitions().asScala
+ )
+ rdd.map(ir => {
+ new GenericRowWithSchema(ir.toSeq(requiredSchema).toArray, requiredSchema)
+ })
+ }
+
+
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ data
+ .write
+ .format("phoenix")
+ .option(PhoenixDataSource.TABLE, params(PhoenixDataSource.TABLE))
+ .option(PhoenixDataSource.JDBC_URL, PhoenixDataSource.getJdbcUrlFromOptions(dataSourceOptions))
Review Comment:
Yes, the PhoenixDataSource.getJdbcUrlFromOptions method handle passing jdbcUrl, zkUrl and also when none is specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736634460
##########
phoenix5-spark/README.md:
##########
@@ -89,7 +147,7 @@ public class PhoenixSparkRead {
### Save DataFrames to Phoenix using DataSourceV2
The `save` is method on DataFrame allows passing in a data source type. You can use
-`phoenix` for DataSourceV2 and must also pass in a `table` and `zkUrl` parameter to
+`phoenix` for DataSourceV2 and must also pass in a `table` and `jdbcUrl` parameter to
Review Comment:
Done.
##########
phoenix5-spark/README.md:
##########
@@ -141,32 +201,32 @@ public class PhoenixSparkWriteFromInputTable {
public static void main() throws Exception {
SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("phoenix-test")
- .set("spark.hadoopRDD.ignoreEmptySplits", "false");
- JavaSparkContext jsc = new JavaSparkContext(sparkConf);
- SQLContext sqlContext = new SQLContext(jsc);
+ .set("spark.hadoopRDD.ignoreEmptySplits", "false");
+ SparkSessinon spark = SparkSession.builder().config(sparkConf).getOrCreate();
// Load INPUT_TABLE
- Dataset<Row> df = sqlContext
+ Dataset<Row> df = spark
.read()
.format("phoenix")
.option("table", "INPUT_TABLE")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
.load();
// Save to OUTPUT_TABLE
df.write()
.format("phoenix")
.mode(SaveMode.Overwrite)
.option("table", "OUTPUT_TABLE")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
.save();
- jsc.stop();
}
}
```
### Save from an external RDD with a schema to a Phoenix table
Just like the previous example, you can pass in the data source type as `phoenix` and specify the `table` and
-`zkUrl` parameters indicating which table and server to persist the DataFrame to.
+`jdbcUrl` parameters indicating which table and server to persist the DataFrame to.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2318576509
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 9s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 10m 27s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 7s | root in master failed. |
| -1 :x: | compile | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 6s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 28s | master passed |
| -1 :x: | javadoc | 0m 6s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 7s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 6s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 7s | phoenix5-spark3 in master failed. |
| -0 :warning: | patch | 0m 50s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 5s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 5s | root in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 7s | phoenix5-spark: The patch generated 15 new + 36 unchanged - 74 fixed = 51 total (was 110) |
| -1 :x: | checkstyle | 0m 8s | phoenix5-spark3: The patch generated 18 new + 40 unchanged - 80 fixed = 58 total (was 120) |
| -1 :x: | markdownlint | 0m 2s | The patch generated 38 new + 97 unchanged - 15 fixed = 135 total (was 112) |
| -1 :x: | whitespace | 0m 0s | The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 6s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 6s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 6s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 13s | The patch generated 4 ASF License warnings. |
| | | 14m 20s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 1ecc1937d3fe 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / a1c298c |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 52 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/5/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1735668680
##########
phoenix5-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkDatasourceV1ITTenantSpecific.scala:
##########
@@ -105,39 +102,6 @@ class PhoenixSparkITTenantSpecific extends AbstractPhoenixSparkIT {
verifyResults
}
- ignore("Can write a DataFrame using 'DataFrame.write' to tenant-specific view - Spark2 sparse columns") {
Review Comment:
This class is for datasourceV1 multitenant tests. And this test was a datasource V2 test.
I faced issues with multitenant table. I rewrote this test and all other multitenant tests in the PhoenixSparkITTenantSpecific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql for spark2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2295288151
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 10m 58s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 9s | root in master failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 33s | master passed |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark3 in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 6s | root in the patch failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 8s | phoenix5-spark: The patch generated 15 new + 22 unchanged - 73 fixed = 37 total (was 95) |
| -1 :x: | checkstyle | 0m 10s | phoenix5-spark3: The patch generated 19 new + 38 unchanged - 81 fixed = 57 total (was 119) |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 17s | The patch generated 4 ASF License warnings. |
| | | 22m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 1dbaf2f28949 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / a1c298c |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/1/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2302576368
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 10m 47s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 7s | root in master failed. |
| -1 :x: | compile | 0m 6s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 6s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 27s | master passed |
| -1 :x: | javadoc | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 7s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 6s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 7s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 6s | phoenix5-spark3 in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 6s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 4s | root in the patch failed. |
| -1 :x: | compile | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 7s | phoenix5-spark: The patch generated 15 new + 23 unchanged - 73 fixed = 38 total (was 96) |
| -1 :x: | checkstyle | 0m 7s | phoenix5-spark3: The patch generated 19 new + 39 unchanged - 81 fixed = 58 total (was 120) |
| -1 :x: | markdownlint | 0m 4s | The patch generated 42 new + 101 unchanged - 11 fixed = 143 total (was 112) |
| -1 :x: | whitespace | 0m 0s | The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 6s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 7s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 6s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 7s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 6s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 7s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 7s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 15s | The patch generated 4 ASF License warnings. |
| | | 17m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 8a64f5cf820e 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / a1c298c |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 52 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/3/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2343965680
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 21s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 13m 58s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 9s | root in master failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 33s | master passed |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark3 in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 6s | root in the patch failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark: The patch generated 15 new + 35 unchanged - 74 fixed = 50 total (was 109) |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark3: The patch generated 18 new + 39 unchanged - 80 fixed = 57 total (was 119) |
| -1 :x: | markdownlint | 0m 3s | The patch generated 40 new + 95 unchanged - 17 fixed = 135 total (was 112) |
| -1 :x: | whitespace | 0m 0s | The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 17s | The patch generated 4 ASF License warnings. |
| | | 25m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 559bb5f7508f 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 6b6a85b |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/6/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1754926199
##########
phoenix5-spark/README.md:
##########
@@ -118,15 +181,16 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "INPUT_TABLE"))
+ .option("table", "INPUT_TABLE")
.load
// Save to OUTPUT_TABLE
df.write
.format("phoenix")
- .mode(SaveMode.Overwrite)
- .options(Map("table" -> "OUTPUT_TABLE"))
+ .mode(SaveMode.Append)
Review Comment:
I made a mistake when duplicating examples, spark2 connector only support Overwrite mode, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1754900264
##########
phoenix5-spark/README.md:
##########
@@ -45,12 +57,14 @@ val spark = SparkSession
val df = spark.sqlContext
.read
.format("phoenix")
- .options(Map("table" -> "TABLE1"))
+ .option("table", "TABLE1")
Review Comment:
I don't think that there is a preferred method. Spark doc use .option() in their examples.
option can be called multiple times. (Key,value) pair are added to spark DataFrameReader or DataFrameWriter `extraOptions` each time the method is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2355453904
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 13m 51s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 8s | root in master failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 33s | master passed |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 6s | root in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark: The patch generated 15 new + 35 unchanged - 74 fixed = 50 total (was 109) |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark3: The patch generated 18 new + 39 unchanged - 80 fixed = 57 total (was 119) |
| -1 :x: | markdownlint | 0m 4s | The patch generated 40 new + 95 unchanged - 17 fixed = 135 total (was 112) |
| -1 :x: | whitespace | 0m 0s | The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 17s | The patch generated 4 ASF License warnings. |
| | | 19m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 576a826c70d9 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / cd2fcf0 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/9/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2354515980
It's a major pain to identify any problem from the 40MB scalatest output.
It would be really nice if there was a newer version that split its outputs failsafe/maven style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2306963632
@stoty, can you review please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql for spark2 and spark3 [phoenix-connectors]
Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#issuecomment-2295328672
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 11m 4s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 9s | root in master failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in master failed. |
| +1 :green_heart: | checkstyle | 0m 33s | master passed |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark3 in master failed. |
| -1 :x: | scaladoc | 0m 8s | phoenix5-spark in master failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in master failed. |
| -1 :x: | spotbugs | 0m 9s | phoenix5-spark in master failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 6s | root in the patch failed. |
| -1 :x: | compile | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | compile | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | javac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | javac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scalac | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | scalac | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark: The patch generated 15 new + 22 unchanged - 73 fixed = 37 total (was 95) |
| -1 :x: | checkstyle | 0m 9s | phoenix5-spark3: The patch generated 19 new + 38 unchanged - 81 fixed = 57 total (was 119) |
| -1 :x: | markdownlint | 0m 4s | The patch generated 38 new + 87 unchanged - 25 fixed = 125 total (was 112) |
| -1 :x: | whitespace | 0m 1s | The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | javadoc | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | javadoc | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark in the patch failed. |
| -1 :x: | scaladoc | 0m 9s | phoenix5-spark3 in the patch failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | spotbugs | 0m 8s | phoenix5-spark3 in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 8s | phoenix5-spark in the patch failed. |
| -1 :x: | unit | 0m 8s | phoenix5-spark3 in the patch failed. |
| -1 :x: | asflicense | 0m 17s | The patch generated 4 ASF License warnings. |
| | | 16m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/139 |
| Optional Tests | dupname asflicense markdownlint javac javadoc unit spotbugs hbaseanti checkstyle compile scalac scaladoc |
| uname | Linux 75d5c75446da 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / a1c298c |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-compile-phoenix5-spark3.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/branch-spotbugs-phoenix5-spark3.txt |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-compile-phoenix5-spark.txt |
| scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-compile-phoenix5-spark3.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark.txt |
| javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-javadoc-phoenix5-spark3.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark.txt |
| scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-scaladoc-phoenix5-spark3.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark.txt |
| spotbugs | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-spotbugs-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-unit-phoenix5-spark.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-unit-phoenix5-spark3.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: phoenix5-spark phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-139/2/console |
| versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736635064
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/PhoenixRDD.scala:
##########
@@ -14,14 +14,14 @@
package org.apache.phoenix.spark
import java.sql.DriverManager
-
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.HConstants
import org.apache.hadoop.io.NullWritable
import org.apache.phoenix.jdbc.PhoenixDriver
import org.apache.phoenix.mapreduce.PhoenixInputFormat
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil
import org.apache.phoenix.query.HBaseFactoryProvider
+import org.apache.phoenix.spark.datasource.v2.SparkSchemaUtil
Review Comment:
Done
##########
phoenix5-spark/src/main/scala/org/apache/phoenix/spark/PhoenixRelation.scala:
##########
@@ -18,8 +18,9 @@
package org.apache.phoenix.spark
import org.apache.hadoop.conf.Configuration
+import org.apache.phoenix.spark.datasource.v2.FilterExpressionCompiler
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736613524
##########
phoenix5-spark/README.md:
##########
@@ -249,14 +307,13 @@ public class PhoenixSparkWriteFromRDDWithSchema {
}
// Create a DataFrame from the rows and the specified schema
- df = spark.createDataFrame(rows, schema);
+ Dataset<Row> df = spark.createDataFrame(rows, schema);
df.write()
.format("phoenix")
.mode(SaveMode.Overwrite)
.option("table", "OUTPUT_TABLE")
+ .option("jdbcUrl", "jdbc:phoenix:zkHost:zkport")
.save();
-
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736661755
##########
phoenix5-spark/src/it/resources/globalSetup.sql:
##########
@@ -60,7 +60,7 @@ UPSERT INTO "small" VALUES ('key1', 'foo', 10000)
UPSERT INTO "small" VALUES ('key2', 'bar', 20000)
UPSERT INTO "small" VALUES ('key3', 'xyz', 30000)
-CREATE TABLE MULTITENANT_TEST_TABLE (TENANT_ID VARCHAR NOT NULL, ORGANIZATION_ID VARCHAR, GLOBAL_COL1 VARCHAR CONSTRAINT pk PRIMARY KEY (TENANT_ID, ORGANIZATION_ID)) MULTI_TENANT=true
+CREATE TABLE MULTITENANT_TEST_TABLE (TENANT_ID VARCHAR NOT NULL, ORGANIZATION_ID VARCHAR, GLOBAL_COL1 VARCHAR CONSTRAINT pk PRIMARY KEY (TENANT_ID, ORGANIZATION_ID)) MULTI_TENANT=true, SALT_BUCKETS = 20
Review Comment:
I added a new multitenant table with salt to test multitenant and salted case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6783 Add spark-sql support for spark2 datasourceV2 and spark3 [phoenix-connectors]
Posted by "rejeb (via GitHub)" <gi...@apache.org>.
rejeb commented on code in PR #139:
URL: https://github.com/apache/phoenix-connectors/pull/139#discussion_r1736626191
##########
phoenix5-spark/src/it/java/org/apache/phoenix/spark/OrderByIT.java:
##########
@@ -46,15 +46,9 @@
import org.apache.spark.sql.sources.v2.DataSourceOptions;
import org.junit.Test;
import org.junit.experimental.categories.Category;
-import org.junit.Ignore;
-
-
-
-import scala.Option;
-import scala.collection.JavaConverters;
@Category(ParallelStatsDisabledTest.class)
-public class OrderByIT extends BaseOrderByIT {
Review Comment:
Bad extends, BaseOrderByIT runs Phoenix based IT tests. These tests are out of context here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org