You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by rahulforallp <gi...@git.apache.org> on 2017/12/05 14:20:55 UTC
[GitHub] carbondata pull request #1616: [WIP] Code refactored
GitHub user rahulforallp opened a pull request:
https://github.com/apache/carbondata/pull/1616
[WIP] Code refactored
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rahulforallp/incubator-carbondata code_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1616.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1616
----
commit e4cae893d762edd2e5ac2b934a6c9a96a21c09a2
Author: rahulforallp <ra...@knoldus.in>
Date: 2017-12-05T14:18:38Z
[WIP] Code refactored
----
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851] Code refactored for better ...
Posted by rahulforallp <gi...@git.apache.org>.
Github user rahulforallp closed the pull request at:
https://github.com/apache/carbondata/pull/1616
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/635/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/546/
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851][WIP] Code refactored for be...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1616#discussion_r155471634
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -151,58 +136,76 @@ object CarbonSession {
return session
}
- // No active nor global default session. Create a new one.
- val sparkContext = userSuppliedContext.getOrElse {
- // set app name if not given
- val randomAppName = java.util.UUID.randomUUID().toString
- val sparkConf = new SparkConf()
- options.foreach { case (k, v) => sparkConf.set(k, v) }
- if (!sparkConf.contains("spark.app.name")) {
- sparkConf.setAppName(randomAppName)
+ // Global synchronization so we will only set the default session once.
+ SparkSession.synchronized {
+ // If the current thread does not have an active session, get it from the global session.
+ session = SparkSession.getDefaultSession match {
+ case Some(sparkSession: CarbonSession) =>
+ if ((sparkSession ne null) && !sparkSession.sparkContext.isStopped) {
+ options
+ .foreach { case (k, v) => sparkSession.sessionState.conf.setConfString(k, v) }
+ sparkSession
+ } else {
+ null
+ }
+ case _ => null
}
- val sc = SparkContext.getOrCreate(sparkConf)
- // maybe this is an existing SparkContext, update its SparkConf which maybe used
- // by SparkSession
- options.foreach { case (k, v) => sc.conf.set(k, v) }
- if (!sc.conf.contains("spark.app.name")) {
- sc.conf.setAppName(randomAppName)
+ if (session ne null) {
+ return session
}
- sc
- }
- session = new CarbonSession(sparkContext)
- val carbonProperties = CarbonProperties.getInstance()
- if (storePath != null) {
- carbonProperties.addProperty(CarbonCommonConstants.STORE_LOCATION, storePath)
- // In case if it is in carbon.properties for backward compatible
- } else if (carbonProperties.getProperty(CarbonCommonConstants.STORE_LOCATION) == null) {
- carbonProperties.addProperty(CarbonCommonConstants.STORE_LOCATION,
- session.sessionState.conf.warehousePath)
- }
- options.foreach { case (k, v) => session.sessionState.conf.setConfString(k, v) }
- SparkSession.setDefaultSession(session)
- try {
- CommonUtil.cleanInProgressSegments(
- carbonProperties.getProperty(CarbonCommonConstants.STORE_LOCATION), sparkContext)
- } catch {
- case e: Throwable =>
- // catch all exceptions to avoid CarbonSession initialization failure
- LogServiceFactory.getLogService(this.getClass.getCanonicalName)
- .error(e, "Failed to clean in progress segments")
- }
- // Register a successfully instantiated context to the singleton. This should be at the
- // end of the class definition so that the singleton is updated only if there is no
- // exception in the construction of the instance.
- sparkContext.addSparkListener(new SparkListener {
- override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
- SparkSession.setDefaultSession(null)
- SparkSession.sqlListener.set(null)
+ // No active nor global default session. Create a new one.
+ val sparkContext = userSuppliedContext.getOrElse {
+ // set app name if not given
+ val randomAppName = java.util.UUID.randomUUID().toString
+ val sparkConf = new SparkConf()
+ options.foreach { case (k, v) => sparkConf.set(k, v) }
+ if (!sparkConf.contains("spark.app.name")) {
+ sparkConf.setAppName(randomAppName)
+ }
+ val sc = SparkContext.getOrCreate(sparkConf)
+ // maybe this is an existing SparkContext, update its SparkConf which maybe used
+ // by SparkSession
+ options.foreach { case (k, v) => sc.conf.set(k, v) }
+ if (!sc.conf.contains("spark.app.name")) {
+ sc.conf.setAppName(randomAppName)
+ }
+ sc
}
- })
- session.streams.addListener(new CarbonStreamingQueryListener(session))
- }
- session
+ session = new CarbonSession(sparkContext)
--- End diff --
Why these changes
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Refactor to use only SegmentsToAcc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1890/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1829/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1838/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/604/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/471/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Refactor to use only SegmentsToAcc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1888/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1735/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/609/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1822/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/584/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by rahulforallp <gi...@git.apache.org>.
Github user rahulforallp commented on the issue:
https://github.com/apache/carbondata/pull/1616
retest this please
---
[GitHub] carbondata issue #1616: [WIP] Code refactored
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/466/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/657/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1803/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1616
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2147/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1616
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2114/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1837/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1853/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Refactor to use only SegmentsToAcc...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on the issue:
https://github.com/apache/carbondata/pull/1616
retest this please
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/599/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by rahulforallp <gi...@git.apache.org>.
Github user rahulforallp commented on the issue:
https://github.com/apache/carbondata/pull/1616
retest this please
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851][WIP] Code refactored for be...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1616#discussion_r155466219
--- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -990,4 +990,11 @@ public boolean isCarbonProperty(String key) {
return addedProperty;
}
+ /**
+ *
+ * @param externalProperties
+ */
+ public void addPropertyToPropertSet(Set<String> externalProperties) {
--- End diff --
'y' missing
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Refactor to use only SegmentsToAcc...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on the issue:
https://github.com/apache/carbondata/pull/1616
LGTM
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/613/
---
[GitHub] carbondata pull request #1616: [WIP] Code refactored
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1616#discussion_r154959616
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
@@ -121,6 +121,8 @@
private boolean hasDataMapSchema;
+ private Map<String, Object> propertiesMap;
--- End diff --
There is a table property map in the TableSchema already, it is in the TableInfo
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851][WIP] Code refactored for be...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1616#discussion_r155470927
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java ---
@@ -126,6 +126,10 @@
private static final String DATA_MAP_DSTR = "mapreduce.input.carboninputformat.datamapdstr";
public static final String DATABASE_NAME = "mapreduce.input.carboninputformat.databaseName";
public static final String TABLE_NAME = "mapreduce.input.carboninputformat.tableName";
+ // to be used in cases where main table status file is not done and newly created segment need to
+ // be accessed
+ public static final String FORCE_ACCESS_SEGMENT =
+ "mapreduce.input.carboninputformat.force.access.segment";
--- End diff --
Dont needs a new flag, use getValidateSegmentsToAccess, mapreduce.input.carboninputformat.validsegments
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1835/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1766/
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851] Refactor to use only Segmen...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1616
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Refactor to use only SegmentsToAcc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/659/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1616
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2195/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/610/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1616
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2201/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by rahulforallp <gi...@git.apache.org>.
Github user rahulforallp commented on the issue:
https://github.com/apache/carbondata/pull/1616
retest this please
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/630/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1861/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Refactor to use only SegmentsToAcc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/661/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/596/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1841/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by rahulforallp <gi...@git.apache.org>.
Github user rahulforallp commented on the issue:
https://github.com/apache/carbondata/pull/1616
retest this please
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1859/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1770/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1864/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/505/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/632/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1616
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2196/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1616
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2119/
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851][WIP] Code refactored for be...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1616#discussion_r155478396
--- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java ---
@@ -196,8 +212,13 @@ public static void deleteStorePath(String path) {
*/
public static void deleteLocalDataLoadFolderLocation(CarbonLoadModel loadModel,
boolean isCompactionFlow, boolean isAltPartitionFlow) {
- String databaseName = loadModel.getDatabaseName();
String tableName = loadModel.getTableName();
+ deleteLocalDataLoadFolderLocation(loadModel, isCompactionFlow, isAltPartitionFlow, tableName);
--- End diff --
Define TableProcessingOperations Class, mark it as developer api, so that it is not modified.
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1832/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/625/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1826/
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851] Code refactored for better ...
Posted by rahulforallp <gi...@git.apache.org>.
GitHub user rahulforallp reopened a pull request:
https://github.com/apache/carbondata/pull/1616
[CARBONDATA-1851] Code refactored for better usablity to external system
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed? **user interface is not changed.**
- [X] Any backward compatibility impacted? **No**
- [X] Document update required? **No**
- [X] Testing done
This PR contains **refactored code** . So no new test-case required.
Only UT, FT and SDV test success report required from CI is sufficient.
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rahulforallp/incubator-carbondata code_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1616.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1616
----
commit 98256151b1dae6071332007ee5c60f51702accf2
Author: rahulforallp <ra...@knoldus.in>
Date: 2017-12-05T14:18:38Z
Code refactored
----
---
[GitHub] carbondata pull request #1616: [CARBONDATA-1851][WIP] Code refactored for be...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1616#discussion_r155472604
--- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java ---
@@ -132,10 +133,25 @@ public boolean accept(CarbonFile file) {
public static void deletePartialLoadDataIfExist(CarbonLoadModel loadModel,
final boolean isCompactionFlow) throws IOException {
CarbonTable carbonTable = loadModel.getCarbonDataLoadSchema().getCarbonTable();
- String metaDataLocation = carbonTable.getMetaDataFilepath();
- final LoadMetadataDetails[] details = SegmentStatusManager.readLoadMetadata(metaDataLocation);
+ deletePartialLoad(loadModel, carbonTable, isCompactionFlow);
+ }
+
+ public static void deletePartialLoad(CarbonLoadModel loadModel, CarbonTable carbonTable,
--- End diff --
Use same name, overloaded function deletePartialLoadDataIfExist
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/508/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851][WIP] Code refactored for better us...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1616
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/582/
---
[GitHub] carbondata issue #1616: [CARBONDATA-1851] Code refactored for better usablit...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1616
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2214/
---