You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/02/02 08:23:44 UTC
[GitHub] [carbondata] areyouokfreejoe opened a new pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
areyouokfreejoe opened a new pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086
Why is this PR needed?
Currently successful load and insert sql return empty Seq in carbondata, we need it to return the segment ID.
What changes were proposed in this PR?
Successful load and insert will return segment ID.
Does this PR introduce any user interface change?
No
Yes. (Successful load and insert will return segment ID.)
Is any new testcase added?
Yes
This closes #4115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770666940
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3627/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-778248958
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3701/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-789372977
LGTM. Thanks for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770666065
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5387/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770408788
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5385/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r569988736
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
In JDBC case, it will return like this
+------------+
|Segment ID|
+------------+
| 0 |
+------------+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r569985571
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
Ok, I have it fixed and re push.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771321901
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3639/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-778251652
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5464/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-773140387
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3668/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771560715
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771456273
> If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
> If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
If we enable 'AUTO_LOAD_MERGE'，then we return and show the segment id before compaction since the user would focus on his load operation. Test case has been added. Please review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r568528970
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -82,6 +83,43 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ protected def createTestTable(tableName: String): Unit = {
+ sql(
+ s"""
Review comment:
Instead of adding new testcase ,
In one of the existing testcases of **loading, insert into, partition table loading, partition table insert into**, just add a validation for segment id
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
+ Seq(Row(loadResultForReturn.getLoadName))
+ } else {
+ rowsForReturn
Review comment:
why are you returning number of rows instead of segment id here ? when will the code enter here when load is success ?
can you add some comment ?
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
I think code is not formatted,
we follow space after if and !=
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
possible to return text like "Successfully loaded to segment id : 1" instead of returning just "1"
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##########
@@ -191,7 +196,15 @@ case class CarbonLoadDataCommand(databaseNameOp: Option[String],
throw ex
}
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
@QiangCai, @ydvpankaj99 : why our checkstyle is not catching this format issues ?
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##########
@@ -191,7 +196,15 @@ case class CarbonLoadDataCommand(databaseNameOp: Option[String],
throw ex
}
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
@QiangCai, @ydvpankaj99 : why our checkstyle is not catching this format issues ?
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
+
+ }
+
+ test("Return segment ID after load and insert when auto merge enabled") {
Review comment:
same comment as above
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
+ Seq(Row(loadResultForReturn.getLoadName))
+ } else {
+ rowsForReturn
Review comment:
partition table case, what will be rowsForReturn?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770398116
Add to whitelist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r569991855
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -82,6 +83,43 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ protected def createTestTable(tableName: String): Unit = {
+ sql(
+ s"""
Review comment:
Can you help point out which test case I can add validation? Because there are too many.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771515497
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3645/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770542595
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3626/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r569990919
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
And in spark sql, it will return as just a num string. The retrun is simple and I think in this case the user can easily catch it and use it as a input of next step. I don't think the too many word is needed here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-777598473
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5073/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770943664
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3637/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe removed a comment on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe removed a comment on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770624729
> Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3626/
Hi, I have solved the issue shown in test case, can you please help rerun the CI?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe closed pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe closed pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r568524277
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##########
@@ -191,7 +196,15 @@ case class CarbonLoadDataCommand(databaseNameOp: Option[String],
throw ex
}
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
@QiangCai, @ydvpankaj99 : why our checkstyle is not catching this format issues ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771464559
> > If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
>
> > If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
>
> If we enable 'AUTO_LOAD_MERGE'，then we return and show the segment id before compaction since the user would focus on his load operation. Test case has been added. Please review.
We are showing the segment id because if we need to query on specific segment then this feature will be helpful. But if we will show the segment id before compaction and will query on specific segment then the operation will fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] brijoobopanna commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
brijoobopanna commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-778194248
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-788600301
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3341/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-777596797
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3315/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770739204
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5388/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r578944568
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -82,6 +83,43 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ protected def createTestTable(tableName: String): Unit = {
+ sql(
+ s"""
Review comment:
can you add in TestLoadGeneral/TestLoadDataWithAutoLoadMerge/InsertIntoCarbonTableTestCase. For partition - StandardPartitionTableLoadingTestCase
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771456273
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 edited a comment on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
nihal0107 edited a comment on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771464559
> > If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
>
> > If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
>
> If we enable 'AUTO_LOAD_MERGE'，then we return and show the segment id before compaction since the user would focus on his load operation. Test case has been added. Please review.
We are showing the segment id because if we need to query on specific segment then this feature will be helpful. But if we will show the segment id before compaction and will query on specific segment then the operation will fail. Better to take the opinion in community.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771385539
If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-787672933
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3758/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r569985309
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
+ Seq(Row(loadResultForReturn.getLoadName))
+ } else {
+ rowsForReturn
Review comment:
It's not number of rows.
It's segment id from partition table case.
Ok, Comment has been added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-773127593
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5428/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-787768063
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3760/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771616575
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5409/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770542291
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5386/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770859314
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3634/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771609869
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3649/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-787669396
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5523/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771385539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771458076
> If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771518487
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5405/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770854884
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5394/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770409051
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3625/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-781563381
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5082/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-788593496
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5100/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r568528970
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -82,6 +83,43 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ protected def createTestTable(tableName: String): Unit = {
+ sql(
+ s"""
Review comment:
Instead of adding new testcase ,
In one of the existing testcases of **loading, insert into, partition table loading, partition table insert into**, just add a validation for segment id
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
+ Seq(Row(loadResultForReturn.getLoadName))
+ } else {
+ rowsForReturn
Review comment:
why are you returning number of rows instead of segment id here ? when will the code enter here when load is success ?
can you add some comment ?
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
I think code is not formatted,
we follow space after if and !=
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
possible to return text like "Successfully loaded to segment id : 1" instead of returning just "1"
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##########
@@ -191,7 +196,15 @@ case class CarbonLoadDataCommand(databaseNameOp: Option[String],
throw ex
}
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
@QiangCai, @ydvpankaj99 : why our checkstyle is not catching this format issues ?
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##########
@@ -191,7 +196,15 @@ case class CarbonLoadDataCommand(databaseNameOp: Option[String],
throw ex
}
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
@QiangCai, @ydvpankaj99 : why our checkstyle is not catching this format issues ?
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
+
+ }
+
+ test("Return segment ID after load and insert when auto merge enabled") {
Review comment:
same comment as above
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
+ Seq(Row(loadResultForReturn.getLoadName))
+ } else {
+ rowsForReturn
Review comment:
partition table case, what will be rowsForReturn?
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##########
@@ -191,7 +196,15 @@ case class CarbonLoadDataCommand(databaseNameOp: Option[String],
throw ex
}
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
@QiangCai, @ydvpankaj99 : why our checkstyle is not catching this format issues ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r569988736
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
In JDBC case, it will return like this
+------------+
|Segment ID|
+------------+
| 0|
+------------+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770187158
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on a change in pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on a change in pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#discussion_r569985309
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
+ Seq(Row(loadResultForReturn.getLoadName))
+ } else {
+ rowsForReturn
Review comment:
It's not number of rows.
It's segment id from partition table case.
Ok, Comment has been added.
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -276,7 +280,15 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
}
throw ex
}
- Seq.empty
+ if(loadResultForReturn!=null && loadResultForReturn.getLoadName!=null) {
Review comment:
Ok, I have it fixed and re push.
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
In JDBC case, it will return like this
+------------+
|Segment ID|
+------------+
| 0|
+------------+
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
And in spark sql, it will return as just a num string. The retrun is simple and I think in this case the user can easily catch it and use it as a input of next step. I don't think the too many word is needed here.
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -100,6 +138,56 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
private lazy val location = CarbonProperties.getStorePath()
+ test("Return segment ID after load and insert") {
+ val tableName = "test_table"
+ val inputTableName = "csv_table"
+ val inputPath = s"$resourcesPath/data_alltypes.csv"
+ dropTable(tableName)
+ dropTable(inputTableName)
+ createAndLoadInputTable(inputTableName, inputPath)
+ createTestTable(tableName)
+ checkAnswer(sql(
+ s"""
+ | INSERT INTO TABLE $tableName
+ | SELECT shortField, intField, bigintField, doubleField, stringField,
+ | from_unixtime(unix_timestamp(timestampField,'yyyy/M/dd')) timestampField, decimalField,
+ | cast(to_date(from_unixtime(unix_timestamp(dateField,'yyyy/M/dd'))) as date), charField
+ | FROM $inputTableName
+ """.stripMargin), Seq(Row("0")))
+ checkAnswer(sql(
+ s"LOAD DATA LOCAL INPATH '$inputPath'" +
+ s" INTO TABLE $tableName" +
+ " OPTIONS('FILEHEADER'=" +
+ "'shortField,intField,bigintField,doubleField,stringField," +
+ "timestampField,decimalField,dateField,charField')"), Seq(Row("1")))
Review comment:
In JDBC case, it will return like this
+------------+
|Segment ID|
+------------+
| 0 |
+------------+
##########
File path: integration/spark/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala
##########
@@ -82,6 +83,43 @@ class CarbonCommandSuite extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ protected def createTestTable(tableName: String): Unit = {
+ sql(
+ s"""
Review comment:
Can you help point out which test case I can add validation? Because there are too many.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 edited a comment on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
nihal0107 edited a comment on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771464559
> > If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
>
> > If we enable the property `ENABLE_AUTO_LOAD_MERGE` then which segment id are we planning to show, the segment generated after compaction or before compaction? Better to add a test case for that scenario also.
>
> If we enable 'AUTO_LOAD_MERGE'，then we return and show the segment id before compaction since the user would focus on his load operation. Test case has been added. Please review.
We are showing the segment id because if we need to query on specific segment then this feature will be helpful. But if we will show the segment id before compaction and will query on specific segment then the operation will fail. Better to take the opinion in community.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-787765521
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771318129
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771318129
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5399/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770739979
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3628/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770940187
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5397/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-773127593
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-770624729
> Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3626/
Hi, I have solved the issue shown in test case, can you please help rerun the CI?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771560715
@nihal0107 : As per me, showing the original single segment before compaction is ok, because it is load command and not the compaction command.
so, when load command finishes we can give segment id that loaded and user can do show segments before querying that segment to see if it has undergone compaction or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-771560807
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-781561235
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3325/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] areyouokfreejoe closed pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID
Posted by GitBox <gi...@apache.org>.
areyouokfreejoe closed pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org