You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by QiangCai <gi...@git.apache.org> on 2017/01/04 03:36:14 UTC
[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...
GitHub user QiangCai opened a pull request:
https://github.com/apache/incubator-carbondata/pull/494
[CARBONDATA-218]Using CSVInputFormat instead of spark-csv during dictionary geneartion
1. Using CSVInputFormat instead of spark-csv during dictionary geneartion
2. Remove spark-csv dependency from whole project.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/QiangCai/incubator-carbondata unifyCsvReader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/494.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #494
----
commit 34f55769d74969cff5d9a322ad8ae0cbf14befdc
Author: QiangCai <qi...@qq.com>
Date: 2017-01-03T08:28:06Z
unify csv reader
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/494
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #494: [CARBONDATA-218]Using CSVInputFormat instea...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/incubator-carbondata/pull/494
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/494#discussion_r94544073
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
@@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
*/
def loadDataFrame(sqlContext: SQLContext,
carbonLoadModel: CarbonLoadModel): DataFrame = {
- val df = sqlContext.read
- .format("com.databricks.spark.csv.newapi")
- .option("header", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
- "true"
- } else {
- "false"
- }
- })
- .option("delimiter", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
- "" + DEFAULT_SEPARATOR
- } else {
- carbonLoadModel.getCsvDelimiter
+ val hadoopConfiguration = new Configuration()
+ CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
+ hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
--- End diff --
use `FileInputFormat.addInput`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/494#discussion_r94552530
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
@@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
*/
def loadDataFrame(sqlContext: SQLContext,
carbonLoadModel: CarbonLoadModel): DataFrame = {
- val df = sqlContext.read
- .format("com.databricks.spark.csv.newapi")
- .option("header", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
- "true"
- } else {
- "false"
- }
- })
- .option("delimiter", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
- "" + DEFAULT_SEPARATOR
- } else {
- carbonLoadModel.getCsvDelimiter
+ val hadoopConfiguration = new Configuration()
+ CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
+ hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/494#discussion_r94545504
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
@@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
*/
def loadDataFrame(sqlContext: SQLContext,
carbonLoadModel: CarbonLoadModel): DataFrame = {
- val df = sqlContext.read
- .format("com.databricks.spark.csv.newapi")
- .option("header", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
- "true"
- } else {
- "false"
- }
- })
- .option("delimiter", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
- "" + DEFAULT_SEPARATOR
- } else {
- carbonLoadModel.getCsvDelimiter
+ val hadoopConfiguration = new Configuration()
+ CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
+ hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
--- End diff --
FileInputFormat.addInputPath method need a Job type paramter.
In addition, this FactFilePath already consist of all file path, we can directly set input path, no need to separate path and add path again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #494: [CARBONDATA-218]Using CSVInputFormat instea...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/494
Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/429/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/494#discussion_r94552544
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
@@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
*/
def loadDataFrame(sqlContext: SQLContext,
carbonLoadModel: CarbonLoadModel): DataFrame = {
- val df = sqlContext.read
- .format("com.databricks.spark.csv.newapi")
- .option("header", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
- "true"
- } else {
- "false"
- }
- })
- .option("delimiter", {
- if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
- "" + DEFAULT_SEPARATOR
- } else {
- carbonLoadModel.getCsvDelimiter
+ val hadoopConfiguration = new Configuration()
+ CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
+ hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---