You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2022/06/06 11:24:18 UTC
[GitHub] [carbondata] maheshrajus opened a new pull request, #4276: [WIP] dropped partition data moving to trash
maheshrajus opened a new pull request, #4276:
URL: https://github.com/apache/carbondata/pull/4276
### Why is this PR needed?
todo
### What changes were proposed in this PR?
todo
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1184540573
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/786/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1150189737
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1150581100
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165624027
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1168638934
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918539981
##########
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java:
##########
@@ -268,4 +268,45 @@ public static String getCompleteTrashFolderPath(String tablePath, long timeStamp
timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR + CarbonTablePath
.SEGMENT_PREFIX + segmentNumber;
}
+
+ /**
+ * This will give the complete path of the trash folder with the timestamp and the partition name
+ *
+ * @param tablePath absolute table path
+ * @param timeStampSubFolder the timestamp for the clean files operation
+ * @param partitionName partition name for which files are moved to the trash folder
+ */
+ public static String getCompleteTrashFolderPathForPartition(String tablePath,
+ long timeStampSubFolder, String partitionName) {
+ return CarbonTablePath.getTrashFolderPath(tablePath) + CarbonCommonConstants.FILE_SEPARATOR
+ + timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR
+ + partitionName;
+ }
+
+ /**
+ * The below method copies dropped partition files to the trash folder.
+ *
+ * @param filesToCopy absolute path of the files to copy to the trash folder
+ * @param trashFolderWithTimestamp trashfolderpath with complete timestamp and segment number
+ */
+ public static void copyFilesToTrash(String filesToCopy, String trashFolderWithTimestamp) {
+ try {
+ if (!FileFactory.isFileExist(trashFolderWithTimestamp)) {
+ FileFactory.mkdirs(trashFolderWithTimestamp);
+ }
+ // check if file exists before copying
+ if (FileFactory.isFileExist(filesToCopy)) {
+ CarbonFile folder = FileFactory.getCarbonFile(filesToCopy);
+ CarbonFile[] dataFiles = folder.listFiles();
+ for (CarbonFile carbonFile : dataFiles) {
+ copyFileToTrashFolder(carbonFile.getAbsolutePath(), trashFolderWithTimestamp);
+ }
+ } else {
+ LOGGER.warn("Folder not copied to trash as partition folder does not exist");
+ }
+ } catch (IOException e) {
+ // If file is already moved or not found then continue with other files
+ LOGGER.warn("Unable to copy file to trash folder as file already moved or not found.", e);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918548519
##########
core/src/main/java/org/apache/carbondata/core/util/CleanFilesUtil.java:
##########
@@ -206,10 +206,12 @@ private static void getStaleSegmentFiles(CarbonTable carbonTable, List<String> s
/**
* This method will delete all the empty partition folders starting from the table path
*/
- private static void deleteEmptyPartitionFoldersRecursively(CarbonFile tablePath) {
+ public static void deleteEmptyPartitionFoldersRecursively(CarbonFile tablePath) {
CarbonFile[] listOfFiles = tablePath.listFiles();
if (listOfFiles.length == 0) {
tablePath.delete();
+ // if parent file folder also empty then delete that too.
+ deleteEmptyPartitionFoldersRecursively(tablePath.getParentFile());
Review Comment:
this i added as part of bug fix. If partition folder is empty after moving to trash, it will delete empty folder. After moving if database folder also empty then it will delete that too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r920739192
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
+ }
+ val timeStamp = CarbonUpdateUtil.readCurrentTime();
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1188593741
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1179978396
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165326742
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6367/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1169669510
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/767/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165326724
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r909218839
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +106,25 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames.head.mkString("/")
Review Comment:
Can we expose a property to enable/disable this operation during drop partition since it can degrade drop partition performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1150033087
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1177244293
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1184120916
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4656/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r921025292
##########
docs/configuration-parameters.md:
##########
@@ -58,6 +58,7 @@ This section provides the details of all the configurations required for the Car
| carbon.lock.class | (none) | This specifies the implementation of ICarbonLock interface to be used for acquiring the locks in case of concurrent operations |
| carbon.data.file.version | V3 | This specifies carbondata file format version. Carbondata file format has evolved with time from V1 to V3 in terms of metadata storage and IO level pruning capabilities. You can find more details [here](https://carbondata.apache.org/file-structure-of-carbondata.html#carbondata-file-format). |
| spark.carbon.hive.schema.store | false | Carbondata currently supports 2 different types of metastores for storing schemas. This property specifies if Hive metastore is to be used for storing and retrieving table schemas |
+| carbon.enable.partitiondata.trash | false | This property is set to enable the feature of moving the deleted partition data to trash when dropped partition command is given
Review Comment:
```suggestion
| carbon.enable.partitiondata.trash | false | This property, when enabled, will move the dropped partition data to trash on ALTER DROP PARTITION operation.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918532569
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
+ }
+ val timeStamp = CarbonUpdateUtil.readCurrentTime();
Review Comment:
suppose if want to replace this System.CurrentMillis with some other in future then need to replace in all places. So i just used method call instead of directly using.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918528536
##########
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java:
##########
@@ -2886,5 +2886,14 @@ private CarbonCommonConstants() {
public static final String CARBON_ENABLE_MULTI_VERSION_TABLE_STATUS_DEFAULT = "false";
+ /**
+ * This flag is set to enable the feature of moving the partition data to trash
+ * when dropped partition command is given.
+ * By default it is enabled if any performance degrade then user should disable this feature.
+ */
+ @CarbonProperty
+ public static final String CARBON_ENABLE_PARTITION_DATA_TRASH =
Review Comment:
@akashrn5 can i disable this feature by default ?. if user want then enable and can use it. in this case we will mention only about moving partiton data to trash in documents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1184067700
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r908267713
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +109,28 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames
Review Comment:
Done and tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1169998463
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4637/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1169998465
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1155043815
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/730/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r909214462
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +106,25 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames.head.mkString("/")
Review Comment:
i think, we should move the partition stale folders under respective trash partition folders. here, you are taking the first dropped partition folder to move other partitions data also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1148831564
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/723/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918022752
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
+ }
+ val timeStamp = CarbonUpdateUtil.readCurrentTime();
+ for (index <- droppedPartitionNames.indices) {
Review Comment:
instead of for loop, if index required, you can use `zipwithindex` function
##########
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java:
##########
@@ -2886,5 +2886,14 @@ private CarbonCommonConstants() {
public static final String CARBON_ENABLE_MULTI_VERSION_TABLE_STATUS_DEFAULT = "false";
+ /**
+ * This flag is set to enable the feature of moving the partition data to trash
+ * when dropped partition command is given.
+ * By default it is enabled if any performance degrade then user should disable this feature.
+ */
+ @CarbonProperty
+ public static final String CARBON_ENABLE_PARTITION_DATA_TRASH =
Review Comment:
update in document file also, also add in FAQ, because here you are saying if impacted in performance, can disable.
##########
core/src/main/java/org/apache/carbondata/core/util/CleanFilesUtil.java:
##########
@@ -206,10 +206,12 @@ private static void getStaleSegmentFiles(CarbonTable carbonTable, List<String> s
/**
* This method will delete all the empty partition folders starting from the table path
*/
- private static void deleteEmptyPartitionFoldersRecursively(CarbonFile tablePath) {
+ public static void deleteEmptyPartitionFoldersRecursively(CarbonFile tablePath) {
CarbonFile[] listOfFiles = tablePath.listFiles();
if (listOfFiles.length == 0) {
tablePath.delete();
+ // if parent file folder also empty then delete that too.
+ deleteEmptyPartitionFoldersRecursively(tablePath.getParentFile());
Review Comment:
this may delete the database also, right? why add this here?
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
Review Comment:
do not hard code, please use org.apache.carbondata.core.constants.CarbonCommonConstants#EQUALS
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
+ }
+ val timeStamp = CarbonUpdateUtil.readCurrentTime();
Review Comment:
please directly use System.CurrentMillis, no need to call method for one line
##########
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java:
##########
@@ -268,4 +268,45 @@ public static String getCompleteTrashFolderPath(String tablePath, long timeStamp
timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR + CarbonTablePath
.SEGMENT_PREFIX + segmentNumber;
}
+
+ /**
+ * This will give the complete path of the trash folder with the timestamp and the partition name
+ *
+ * @param tablePath absolute table path
+ * @param timeStampSubFolder the timestamp for the clean files operation
+ * @param partitionName partition name for which files are moved to the trash folder
+ */
+ public static String getCompleteTrashFolderPathForPartition(String tablePath,
+ long timeStampSubFolder, String partitionName) {
+ return CarbonTablePath.getTrashFolderPath(tablePath) + CarbonCommonConstants.FILE_SEPARATOR
+ + timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR
+ + partitionName;
+ }
+
+ /**
+ * The below method copies dropped partition files to the trash folder.
+ *
+ * @param filesToCopy absolute path of the files to copy to the trash folder
+ * @param trashFolderWithTimestamp trashfolderpath with complete timestamp and segment number
+ */
+ public static void copyFilesToTrash(String filesToCopy, String trashFolderWithTimestamp) {
Review Comment:
why are different methods required for handling partition tables? the existing method does not handle it? if so change the method name specific to the partition table, so that no one reuses it by mistake
##########
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java:
##########
@@ -268,4 +268,45 @@ public static String getCompleteTrashFolderPath(String tablePath, long timeStamp
timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR + CarbonTablePath
.SEGMENT_PREFIX + segmentNumber;
}
+
+ /**
+ * This will give the complete path of the trash folder with the timestamp and the partition name
+ *
+ * @param tablePath absolute table path
+ * @param timeStampSubFolder the timestamp for the clean files operation
+ * @param partitionName partition name for which files are moved to the trash folder
+ */
+ public static String getCompleteTrashFolderPathForPartition(String tablePath,
+ long timeStampSubFolder, String partitionName) {
+ return CarbonTablePath.getTrashFolderPath(tablePath) + CarbonCommonConstants.FILE_SEPARATOR
+ + timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR
+ + partitionName;
+ }
+
+ /**
+ * The below method copies dropped partition files to the trash folder.
+ *
+ * @param filesToCopy absolute path of the files to copy to the trash folder
+ * @param trashFolderWithTimestamp trashfolderpath with complete timestamp and segment number
+ */
+ public static void copyFilesToTrash(String filesToCopy, String trashFolderWithTimestamp) {
+ try {
+ if (!FileFactory.isFileExist(trashFolderWithTimestamp)) {
+ FileFactory.mkdirs(trashFolderWithTimestamp);
+ }
+ // check if file exists before copying
+ if (FileFactory.isFileExist(filesToCopy)) {
+ CarbonFile folder = FileFactory.getCarbonFile(filesToCopy);
+ CarbonFile[] dataFiles = folder.listFiles();
+ for (CarbonFile carbonFile : dataFiles) {
+ copyFileToTrashFolder(carbonFile.getAbsolutePath(), trashFolderWithTimestamp);
+ }
+ } else {
+ LOGGER.warn("Folder not copied to trash as partition folder does not exist");
+ }
+ } catch (IOException e) {
+ // If file is already moved or not found then continue with other files
+ LOGGER.warn("Unable to copy file to trash folder as file already moved or not found.", e);
Review Comment:
better to identify and give proper error messages rather than keeping users in ambiguous state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r920728055
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
+ }
+ val timeStamp = CarbonUpdateUtil.readCurrentTime();
Review Comment:
for time and in this case it will be milliSec only and many places used directly, so use directly only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1184107777
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r920769079
##########
docs/ddl-of-carbondata.md:
##########
@@ -1107,6 +1107,12 @@ Users can specify which columns to include and exclude for local dictionary gene
ALTER TABLE locationTable DROP PARTITION (country = 'US');
```
+ The below flag is set to enable the feature of moving the deleted partition data to trash when dropped partition command is given.
+ ```
+ carbon.enable.partitiondata.trash
+ ```
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1155019597
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4599/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1155018749
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6342/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r908061474
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -30,13 +30,17 @@ import org.apache.spark.sql.parser.CarbonSparkSqlParserUtil
import org.apache.spark.util.AlterTableUtil
import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.constants.CarbonCommonConstants
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165573736
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165678482
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/757/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1154340499
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6341/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r907039904
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +109,28 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames
+ .head
+ .headOption
+ .get
+ TrashUtil.copyFilesToTrash(carbonPartitionsTobeDropped,
+ TrashUtil.getCompleteTrashFolderPath(
Review Comment:
`getCompleteTrashFolderPath` is used to create Path for Non-partition table(appends Segment Prefix), I think, for partition table, we should create Path with Partition folder name
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +109,28 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames
+ .head
+ .headOption
+ .get
+ TrashUtil.copyFilesToTrash(carbonPartitionsTobeDropped,
+ TrashUtil.getCompleteTrashFolderPath(
+ table.getTablePath,
+ CarbonUpdateUtil.readCurrentTime(),
+ droppedPartitionName))
+ // Delete partition folder after copy to trash
+ carbonPartitionsTobeDropped.asScala.foreach(delPartition => {
Review Comment:
Please format the new code changes
##########
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableDropTestCase.scala:
##########
@@ -218,6 +222,129 @@ class StandardPartitionTableDropTestCase extends QueryTest with BeforeAndAfterAl
Seq(Row(0)))
}
+ test("dropping partition with moving data to trash") {
+ sql("drop table if exists dropPartition1")
+ sql(
+ """
+ | CREATE TABLE dropPartition1 (empno int, empname String, designation String,
+ | workgroupcategory int, workgroupcategoryname String, deptno int,
+ | projectjoindate Timestamp, projectenddate Date,attendance int,
+ | utilization int,salary int)
+ | PARTITIONED BY (deptname String,doj Timestamp,projectcode int)
+ | STORED AS carbondata
+ """.stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='Learning')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='configManagement')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='network')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='protocol')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='security')""")
+ val table = CarbonEnv.getCarbonTable(Option("default"), "dropPartition1")(sqlContext
+ .sparkSession)
+ val tablePath = table.getTablePath
+ val deptname = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=Learning")
+ }
+ assert(deptname.length == 0)
+ val configManagement = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=configManagement")
+ }
+ assert(configManagement.length == 0)
+ val network = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=network")
+ }
+ assert(network.length == 0)
+ val protocol = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=protocol")
+ }
+ assert(protocol.length == 0)
+ val security = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=protocol")
+ }
+ assert(security.length == 0)
+ sql("drop table if exists dropPartition1")
Review Comment:
please add a validation to check the data count after drop partition
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -30,13 +30,17 @@ import org.apache.spark.sql.parser.CarbonSparkSqlParserUtil
import org.apache.spark.util.AlterTableUtil
import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.constants.CarbonCommonConstants
Review Comment:
Please remove unused imports
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +109,28 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames
Review Comment:
Taking Head Option to create Partition folder name in Trash is wrong here. For example, consider a case, where user performs drop multi-level partititon (i.e., `DROP PARTITION(deptname='Learning',doj='2015-12-01 00:00:00' )`) in this case, Trash folder also should have multi-level partition folder, to roll back drop partition if user wants. So, in TRash, please copy under respective multi-level partition folders
##########
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableQueryTestCase.scala:
##########
@@ -494,9 +494,8 @@ test("Creation of partition table should fail if the colname in table schema and
sql(s"""ALTER TABLE staticpartitionlocloadother_new DROP PARTITION(empname='ravi')""")
checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(10)))
sql(s"""alter table staticpartitionlocloadother_new add partition (empname='ravi') location '$location'""")
- checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(20)))
sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE staticpartitionlocloadother_new partition(empname='ravi') OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '"')""")
- checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(30)))
+ checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(20)))
Review Comment:
why this change ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r914672717
##########
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java:
##########
@@ -268,4 +268,44 @@ public static String getCompleteTrashFolderPath(String tablePath, long timeStamp
timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR + CarbonTablePath
.SEGMENT_PREFIX + segmentNumber;
}
+
+ /**
+ * This will give the complete path of the trash folder with the timestamp and the partition name
+ *
+ * @param tablePath absolute table path
+ * @param timeStampSubFolder the timestamp for the clean files operation
+ * @param partitionName partition name for which files are moved to the trash folder
+ */
+ public static String getCompleteTrashFolderPathForPartition(String tablePath,
+ long timeStampSubFolder, String partitionName) {
+ return CarbonTablePath.getTrashFolderPath(tablePath) + CarbonCommonConstants.FILE_SEPARATOR
+ + timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR
+ + CarbonTablePath.DATA_PART_PREFIX + partitionName;
Review Comment:
CarbonTablePath.DATA_PART_PREFIX is not needed to make partition path
##########
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java:
##########
@@ -268,4 +268,44 @@ public static String getCompleteTrashFolderPath(String tablePath, long timeStamp
timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR + CarbonTablePath
.SEGMENT_PREFIX + segmentNumber;
}
+
+ /**
+ * This will give the complete path of the trash folder with the timestamp and the partition name
+ *
+ * @param tablePath absolute table path
+ * @param timeStampSubFolder the timestamp for the clean files operation
+ * @param partitionName partition name for which files are moved to the trash folder
+ */
+ public static String getCompleteTrashFolderPathForPartition(String tablePath,
+ long timeStampSubFolder, String partitionName) {
+ return CarbonTablePath.getTrashFolderPath(tablePath) + CarbonCommonConstants.FILE_SEPARATOR
+ + timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR
+ + CarbonTablePath.DATA_PART_PREFIX + partitionName;
+ }
+
+ /**
+ * The below method copies dropped partition files to the trash folder.
+ *
+ * @param filesToCopy absolute path of the files to copy to the trash folder
+ * @param trashFolderWithTimestamp trashfolderpath with complete timestamp and segment number
+ */
+ public static void copyFilesToTrash(String filesToCopy, String trashFolderWithTimestamp)
+ throws IOException, InterruptedException {
Review Comment:
invalid method exception definition. Please remove
##########
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java:
##########
@@ -268,4 +268,44 @@ public static String getCompleteTrashFolderPath(String tablePath, long timeStamp
timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR + CarbonTablePath
.SEGMENT_PREFIX + segmentNumber;
}
+
+ /**
+ * This will give the complete path of the trash folder with the timestamp and the partition name
+ *
+ * @param tablePath absolute table path
+ * @param timeStampSubFolder the timestamp for the clean files operation
+ * @param partitionName partition name for which files are moved to the trash folder
+ */
+ public static String getCompleteTrashFolderPathForPartition(String tablePath,
+ long timeStampSubFolder, String partitionName) {
+ return CarbonTablePath.getTrashFolderPath(tablePath) + CarbonCommonConstants.FILE_SEPARATOR
+ + timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR
+ + CarbonTablePath.DATA_PART_PREFIX + partitionName;
+ }
+
+ /**
+ * The below method copies dropped partition files to the trash folder.
+ *
+ * @param filesToCopy absolute path of the files to copy to the trash folder
+ * @param trashFolderWithTimestamp trashfolderpath with complete timestamp and segment number
+ */
+ public static void copyFilesToTrash(String filesToCopy, String trashFolderWithTimestamp)
+ throws IOException, InterruptedException {
+ try {
+ if (!FileFactory.isFileExist(trashFolderWithTimestamp)) {
+ FileFactory.mkdirs(trashFolderWithTimestamp);
+ }
+ // check if file exists before copying
+ if (FileFactory.isFileExist(filesToCopy)) {
+ CarbonFile folder = FileFactory.getCarbonFile(filesToCopy);
+ CarbonFile[] dataFiles = folder.listFiles();
+ for (CarbonFile carbonFile : dataFiles) {
+ copyFileToTrashFolder(carbonFile.getAbsolutePath(), trashFolderWithTimestamp);
+ }
+ }
Review Comment:
can log the warning message if the statement `if (FileFactory.isFileExist(filesToCopy)) {` is false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1179771244
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4646/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918533423
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1181479275
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6392/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1179776730
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/777/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1184535449
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r908076649
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +109,28 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames
+ .head
+ .headOption
+ .get
+ TrashUtil.copyFilesToTrash(carbonPartitionsTobeDropped,
+ TrashUtil.getCompleteTrashFolderPath(
Review Comment:
Added separate method for creating trash folder name with given partition folder names
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1168681187
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/763/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1179771062
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6391/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1177235019
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4643/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1177235016
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r920757032
##########
docs/ddl-of-carbondata.md:
##########
@@ -1107,6 +1107,12 @@ Users can specify which columns to include and exclude for local dictionary gene
ALTER TABLE locationTable DROP PARTITION (country = 'US');
```
+ The below flag is set to enable the feature of moving the deleted partition data to trash when dropped partition command is given.
+ ```
+ carbon.enable.partitiondata.trash
+ ```
Review Comment:
please add in configuration md file also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1184122482
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/785/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r909486640
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +106,25 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames.head.mkString("/")
Review Comment:
done
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -105,6 +106,25 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2}
+ }
+ val droppedPartitionName = droppedPartitionNames.head.mkString("/")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918530101
##########
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java:
##########
@@ -268,4 +268,45 @@ public static String getCompleteTrashFolderPath(String tablePath, long timeStamp
timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR + CarbonTablePath
.SEGMENT_PREFIX + segmentNumber;
}
+
+ /**
+ * This will give the complete path of the trash folder with the timestamp and the partition name
+ *
+ * @param tablePath absolute table path
+ * @param timeStampSubFolder the timestamp for the clean files operation
+ * @param partitionName partition name for which files are moved to the trash folder
+ */
+ public static String getCompleteTrashFolderPathForPartition(String tablePath,
+ long timeStampSubFolder, String partitionName) {
+ return CarbonTablePath.getTrashFolderPath(tablePath) + CarbonCommonConstants.FILE_SEPARATOR
+ + timeStampSubFolder + CarbonCommonConstants.FILE_SEPARATOR
+ + partitionName;
+ }
+
+ /**
+ * The below method copies dropped partition files to the trash folder.
+ *
+ * @param filesToCopy absolute path of the files to copy to the trash folder
+ * @param trashFolderWithTimestamp trashfolderpath with complete timestamp and segment number
+ */
+ public static void copyFilesToTrash(String filesToCopy, String trashFolderWithTimestamp) {
Review Comment:
yes, method name changed accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1181479270
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4647/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165475596
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1154214984
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1154664578
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1158827647
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1181498684
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1184499722
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r921026787
##########
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java:
##########
@@ -2886,5 +2886,15 @@ private CarbonCommonConstants() {
public static final String CARBON_ENABLE_MULTI_VERSION_TABLE_STATUS_DEFAULT = "false";
+ /**
+ * This flag is set to enable the feature of moving the partition data to trash
Review Comment:
```suggestion
* Enable this property to move the dropped partition data to trash on ALTER DROP PARTITION operation
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1158974014
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6348/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1169670225
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4636/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1150222497
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/725/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1154482959
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/729/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165414347
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165429954
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/754/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r908062873
##########
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableQueryTestCase.scala:
##########
@@ -494,9 +494,8 @@ test("Creation of partition table should fail if the colname in table schema and
sql(s"""ALTER TABLE staticpartitionlocloadother_new DROP PARTITION(empname='ravi')""")
checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(10)))
sql(s"""alter table staticpartitionlocloadother_new add partition (empname='ravi') location '$location'""")
- checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(20)))
sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE staticpartitionlocloadother_new partition(empname='ravi') OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '"')""")
- checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(30)))
+ checkAnswer(sql(s"select count(deptname) from staticpartitionlocloadother_new"), Seq(Row(20)))
Review Comment:
before we are not removing partition folders so when load again the count was 30. With new change we are removing partition folders when we performed drop partition. as per new change modified the test case and test case is passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r908072157
##########
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableDropTestCase.scala:
##########
@@ -218,6 +222,129 @@ class StandardPartitionTableDropTestCase extends QueryTest with BeforeAndAfterAl
Seq(Row(0)))
}
+ test("dropping partition with moving data to trash") {
+ sql("drop table if exists dropPartition1")
+ sql(
+ """
+ | CREATE TABLE dropPartition1 (empno int, empname String, designation String,
+ | workgroupcategory int, workgroupcategoryname String, deptno int,
+ | projectjoindate Timestamp, projectenddate Date,attendance int,
+ | utilization int,salary int)
+ | PARTITIONED BY (deptname String,doj Timestamp,projectcode int)
+ | STORED AS carbondata
+ """.stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(
+ s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE dropPartition1 OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='Learning')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='configManagement')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='network')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='protocol')""")
+ sql(s"""ALTER TABLE dropPartition1 DROP PARTITION(deptname='security')""")
+ val table = CarbonEnv.getCarbonTable(Option("default"), "dropPartition1")(sqlContext
+ .sparkSession)
+ val tablePath = table.getTablePath
+ val deptname = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=Learning")
+ }
+ assert(deptname.length == 0)
+ val configManagement = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=configManagement")
+ }
+ assert(configManagement.length == 0)
+ val network = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=network")
+ }
+ assert(network.length == 0)
+ val protocol = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=protocol")
+ }
+ assert(protocol.length == 0)
+ val security = FileFactory.getCarbonFile(tablePath).listFiles().filter{
+ file => file.getName.equalsIgnoreCase("deptname=protocol")
+ }
+ assert(security.length == 0)
+ sql("drop table if exists dropPartition1")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1168669257
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4632/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1164252076
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.3/4/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1170025444
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/768/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1165475597
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4624/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r921033456
##########
docs/ddl-of-carbondata.md:
##########
@@ -1107,6 +1107,12 @@ Users can specify which columns to include and exclude for local dictionary gene
ALTER TABLE locationTable DROP PARTITION (country = 'US');
```
+ The below flag is set to enable the feature of moving the deleted partition data to trash when dropped partition command is given.
Review Comment:
better to add it as a note
```suggestion
**NOTE:** Enable [carbon.enable.partitiondata.trash](./configuration-parameters.md#system-configuration) to move dropped partition data to trash during alter table DROP PARTITION.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r918538362
##########
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala:
##########
@@ -106,6 +108,32 @@ case class CarbonAlterTableDropHivePartitionCommand(
ifExists,
purge,
retainData).run(sparkSession)
+ val isPartitionDataTrashEnabled = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH,
+ CarbonCommonConstants.CARBON_ENABLE_PARTITION_DATA_TRASH_DEFAULT).toBoolean
+ if (isPartitionDataTrashEnabled) {
+ // move the partition files to trash folder which are dropped
+ val droppedPartitionNames = partitions.map { partition =>
+ partition.spec.map { specs => specs._1 + "=" + specs._2 }
+ }
+ val timeStamp = CarbonUpdateUtil.readCurrentTime();
+ for (index <- droppedPartitionNames.indices) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a diff in pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on code in PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#discussion_r921037157
##########
docs/faq.md:
##########
@@ -481,4 +482,8 @@ TableStatusRecovery.main(args) --> args is of length two: 1. Database Name 2. Ta
TableStatus Recovery tool cannot recover table status version files for the below two scenarios
1. After compaction, if table status file is lost, cannot recover compacted commit transaction, as the lost version file only has merged load details.
2. After Delete segment by Id/Date, if table status file is lost, cannot recover deleted segment commit transaction, as the lost version file only has the segment status as deleted.
-3. Table status recovery on materialized view table is not supported.
\ No newline at end of file
+3. Table status recovery on materialized view table is not supported.
+
+## Why deleted partition data still showing in file system
+Please enable the feature **moving deleted partition data to trash** by making the `carbon.enable.partitiondata.trash` property true.
Review Comment:
Can mention like below.
By default, the dropped partition data will not be physically removed from the table store until the table is dropped. Enable [carbon.enable.partitiondata.trash](./configuration-parameters.md#system-configuration) property to move all the dropped partitions data to trash during alter table DROP PARTITION operation itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
maheshrajus commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1179746240
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
URL: https://github.com/apache/carbondata/pull/4276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1148624903
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4590/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [WIP] dropped partition data moving to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1148820059
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1158972044
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4276: [CARBONDATA-4338] moving dropped partition data to trash
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on PR #4276:
URL: https://github.com/apache/carbondata/pull/4276#issuecomment-1159010180
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/736/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org