You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/09/23 10:45:05 UTC
[GitHub] [carbondata] ShreelekhyaG opened a new pull request #4222: [WIP] Spatial index creation using spark dataframe
ShreelekhyaG opened a new pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222
### Why is this PR needed?
To support spatial index creation using spark data frame
### What changes were proposed in this PR?
Added spatial properties in carbonOptions and edited existing testcases.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929521520
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4209/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929921110
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/358/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-932082005
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5983/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-936280627
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r718397911
##########
File path: docs/spatial-index-guide.md
##########
@@ -74,6 +74,30 @@ create table source_index(id BIGINT, latitude long, longitude long) stored by 'c
'SPATIAL_INDEX.mygeohash.gridSize'='50',
'SPATIAL_INDEX.mygeohash.conversionRatio'='1000000');
```
+
+Create spatial table using spark dataframe
+
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
Review comment:
assign this to geodf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-940195668
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6046/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-926707593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-930274996
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5962/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-936180779
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6008/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937560507
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-940990329
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/440/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r719427368
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
##########
@@ -82,6 +83,7 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
}
private def makeCreateTableString(schema: StructType, options: CarbonOption): String = {
+ val indexName = options.SPATIAL_INDEX.getOrElse("")
Review comment:
```suggestion
val indexName = options.indexName
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r719175395
##########
File path: docs/spatial-index-guide.md
##########
@@ -74,6 +74,30 @@ create table source_index(id BIGINT, latitude long, longitude long) stored by 'c
'SPATIAL_INDEX.mygeohash.gridSize'='50',
'SPATIAL_INDEX.mygeohash.conversionRatio'='1000000');
```
+
+Create spatial table using spark dataframe
+
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-930275660
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4217/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r717634102
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
Review comment:
the options map is already normalized. so, case insensitive values are also accepted. Edited existing testcase with different cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-925763525
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5947/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r720161624
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
##########
@@ -82,6 +83,7 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
}
private def makeCreateTableString(schema: StructType, options: CarbonOption): String = {
+ val indexName = options.SPATIAL_INDEX.getOrElse("")
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r717632813
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
+ .option("SPATIAL_INDEX.mygeohash.sourcecolumns", "longitude, latitude")
+ .option("SPATIAL_INDEX.mygeohash.originLatitude", "39.832277")
Review comment:
Done
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
+
+ lazy val SPATIAL_INDEX_sourcecolumns: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.sourcecolumns")
+
+ lazy val SPATIAL_INDEX_originLatitude: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.originLatitude")
+
+ lazy val SPATIAL_INDEX_gridSize: Option[String] = options.get("SPATIAL_INDEX.mygeohash.gridSize")
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r717632539
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
Review comment:
Done
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
+ .option("SPATIAL_INDEX.mygeohash.sourcecolumns", "longitude, latitude")
+ .option("SPATIAL_INDEX.mygeohash.originLatitude", "39.832277")
Review comment:
Done
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
+
+ lazy val SPATIAL_INDEX_sourcecolumns: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.sourcecolumns")
+
+ lazy val SPATIAL_INDEX_originLatitude: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.originLatitude")
+
+ lazy val SPATIAL_INDEX_gridSize: Option[String] = options.get("SPATIAL_INDEX.mygeohash.gridSize")
Review comment:
Done
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
Review comment:
the options map is already normalized. so, case insensitive values are also accepted. Edited existing testcase with different cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-936232689
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4262/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ydvpankaj99 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ydvpankaj99 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929974705
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937679314
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4275/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-940203929
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4302/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-936995450
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/411/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-925840490
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938556510
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6036/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934164535
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937560507
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937034773
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r717632539
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937001729
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-925937337
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5948/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-925769644
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4201/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929860057
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-935978537
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-930079200
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/360/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938913829
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6040/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-926685544
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4205/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-925892432
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4202/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929922453
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4212/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-925930301
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/349/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938946896
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938434092
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938548245
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/426/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934360423
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5999/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ydvpankaj99 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ydvpankaj99 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929974705
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929922162
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5958/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934371484
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/398/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r717219729
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
Review comment:
looks like the newly added options are case sensitive. please handle with case insensitive options
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
+ .option("SPATIAL_INDEX.mygeohash.sourcecolumns", "longitude, latitude")
+ .option("SPATIAL_INDEX.mygeohash.originLatitude", "39.832277")
Review comment:
test some options with case insensitve
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
Review comment:
add testcase with invalid options
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
+
+ lazy val SPATIAL_INDEX_sourcecolumns: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.sourcecolumns")
+
+ lazy val SPATIAL_INDEX_originLatitude: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.originLatitude")
+
+ lazy val SPATIAL_INDEX_gridSize: Option[String] = options.get("SPATIAL_INDEX.mygeohash.gridSize")
Review comment:
what if the user doesn't provide some other required options, when spatial index property is provided. Please check and handle
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
##########
@@ -91,7 +101,15 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
"TABLE_PAGE_SIZE_INMB" -> options.tablePageSizeInMb,
"STREAMING" -> Option(options.isStreaming.toString),
"DATEFORMAT" -> options.dateformat,
- "TIMESTAMPFORMAT" -> options.timestampformat
+ "TIMESTAMPFORMAT" -> options.timestampformat,
+ "TIMESTAMPFORMAT" -> options.timestampformat,
Review comment:
duplicate TIMESTAMPFORMAT definition in line 104 and 105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929520144
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/356/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937675168
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6019/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-940976390
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6050/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-939909137
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929860057
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-931939197
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934233445
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934138015
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/394/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934312999
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-930055047
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938522987
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-936760476
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [WIP] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-925761421
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/348/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938608133
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-940212718
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929521054
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5955/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r717219729
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
Review comment:
looks like the newly added options are case sensitive. please handle with case insensitive options
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
+ .option("SPATIAL_INDEX.mygeohash.sourcecolumns", "longitude, latitude")
+ .option("SPATIAL_INDEX.mygeohash.originLatitude", "39.832277")
Review comment:
test some options with case insensitve
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##########
@@ -917,4 +1009,35 @@ class GeoTest extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
sql(s"""LOAD DATA local inpath '$resourcesPath/geodata2.csv' INTO TABLE $tableName OPTIONS
|('DELIMITER'= ',')""".stripMargin)
}
+
+ def createDf(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata.csv")
+ }
+
+ def createDf2(): DataFrame = {
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
+ .csv(s"$resourcesPath/geodata2.csv")
+ }
+
+ def createTableWithDf(geoDf: DataFrame, tableName: String = dfTable1): Unit = {
+ geoDf.write
+ .format("carbondata")
+ .option("tableName", s"$tableName")
+ .option("SPATIAL_INDEX", "mygeohash")
+ .option("SPATIAL_INDEX.mygeohash.type", "geohash")
Review comment:
add testcase with invalid options
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
+
+ lazy val SPATIAL_INDEX_type: Option[String] = options.get("SPATIAL_INDEX.mygeohash.type")
+
+ lazy val SPATIAL_INDEX_sourcecolumns: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.sourcecolumns")
+
+ lazy val SPATIAL_INDEX_originLatitude: Option[String] = options.get(
+ "SPATIAL_INDEX.mygeohash.originLatitude")
+
+ lazy val SPATIAL_INDEX_gridSize: Option[String] = options.get("SPATIAL_INDEX.mygeohash.gridSize")
Review comment:
what if the user doesn't provide some other required options, when spatial index property is provided. Please check and handle
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
##########
@@ -91,7 +101,15 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
"TABLE_PAGE_SIZE_INMB" -> options.tablePageSizeInMb,
"STREAMING" -> Option(options.isStreaming.toString),
"DATEFORMAT" -> options.dateformat,
- "TIMESTAMPFORMAT" -> options.timestampformat
+ "TIMESTAMPFORMAT" -> options.timestampformat,
+ "TIMESTAMPFORMAT" -> options.timestampformat,
Review comment:
duplicate TIMESTAMPFORMAT definition in line 104 and 105
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
Review comment:
please update the document abt this support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929520144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-940802672
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-941001849
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-930100118
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5960/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934126349
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5996/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-929921110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-930261566
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/362/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-931972188
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-935880671
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934353909
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-931944597
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5981/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937694181
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/419/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-938877738
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4295/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-934069382
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-937675168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-932099177
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4238/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-932036908
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#issuecomment-931938698
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4234/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r717459643
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala
##########
@@ -84,5 +84,22 @@ class CarbonOption(options: Map[String, String]) {
lazy val dateformat: Option[String] = options.get("dateformat")
+ lazy val SPATIAL_INDEX: Option[String] = options.get("SPATIAL_INDEX")
Review comment:
please update the document abt this support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4222: [CARBONDATA-4292] Spatial index creation using spark dataframe
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4222:
URL: https://github.com/apache/carbondata/pull/4222#discussion_r718397911
##########
File path: docs/spatial-index-guide.md
##########
@@ -74,6 +74,30 @@ create table source_index(id BIGINT, latitude long, longitude long) stored by 'c
'SPATIAL_INDEX.mygeohash.gridSize'='50',
'SPATIAL_INDEX.mygeohash.conversionRatio'='1000000');
```
+
+Create spatial table using spark dataframe
+
+ val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
+ StructField("longitude", LongType, nullable = false),
+ StructField("latitude", LongType, nullable = false)))
+
+ sqlContext.read.option("delimeter", ",").option("header", "true").schema(geoSchema)
Review comment:
assign this to geodf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org