You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by mohammadshahidkhan <gi...@git.apache.org> on 2016/10/08 21:01:35 UTC
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
GitHub user mohammadshahidkhan opened a pull request:
https://github.com/apache/incubator-carbondata/pull/218
[CARBONDATA-288] In hdfs bad record logger is failing in writing the bad records in log file
**Poblem**
For HDFS file system
CarbonFile logFile = FileFactory.getCarbonFile(filePath, FileType.HDFS);
if filePath does not exits then
Calling CarbonFile.getPath() throws NullPointerException.
**Solution:**
If file does not exist then before accessing the file must be created first.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata badrecord_log_file_writting_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/218.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #218
----
commit 6a9ca5a78bc128f1e4f8c37164fc81dec6b70894
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2016-10-08T20:54:57Z
[CARBONDATA-288] In hdfs bad record logger is failing in writting the bad records
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83015590
--- Diff: processing/src/main/java/org/apache/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java ---
@@ -458,9 +462,11 @@ public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws K
break;
case REDIRECT:
badRecordsLogRedirect = true;
+ badRecordConvertNullDisable= true;
--- End diff --
add space before `=`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/218
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83391717
--- Diff: processing/src/main/java/org/apache/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java ---
@@ -458,9 +462,11 @@ public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws K
break;
case REDIRECT:
badRecordsLogRedirect = true;
+ badRecordConvertNullDisable= true;
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83391617
--- Diff: processing/src/main/java/org/apache/carbondata/processing/surrogatekeysgenerator/csvbased/BadRecordslogger.java ---
@@ -69,9 +68,13 @@
private BufferedWriter bufferedCSVWriter;
private DataOutputStream outCSVStream;
/**
- *
+ * bad record log file path
+ */
+ private String logFilePath;
+ /**
+ * csv file path
*/
- private CarbonFile logFile;
+ private String csvFilePath;
--- End diff --
log file will contains bad record row with the detailed reason of the failure
csv will have only the bad record row.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83014256
--- Diff: integration/spark/src/main/java/org/apache/carbondata/spark/load/CarbonLoadModel.java ---
@@ -117,9 +117,9 @@
private String badRecordsLoggerEnable;
/**
- * defines the option to specify the bad record log redirect to raw csv
+ * defines the option to specify the bad record logger action
*/
- private String badRecordsLoggerRedirect;
+ private String badRecordsLoggerAction;
--- End diff --
This action is not for Logger, right? Perhaps `badRecordsAction` is a better name?
And it should be an enum instead of String
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83014871
--- Diff: processing/src/main/java/org/apache/carbondata/processing/surrogatekeysgenerator/csvbased/BadRecordslogger.java ---
@@ -69,9 +68,13 @@
private BufferedWriter bufferedCSVWriter;
private DataOutputStream outCSVStream;
/**
- *
+ * bad record log file path
+ */
+ private String logFilePath;
+ /**
+ * csv file path
*/
- private CarbonFile logFile;
+ private String csvFilePath;
--- End diff --
What is this csv file? What is the difference from logFilePath?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #218: [CARBONDATA-288] In hdfs bad record ...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83391382
--- Diff: integration/spark/src/main/java/org/apache/carbondata/spark/load/CarbonLoadModel.java ---
@@ -117,9 +117,9 @@
private String badRecordsLoggerEnable;
/**
- * defines the option to specify the bad record log redirect to raw csv
+ * defines the option to specify the bad record logger action
*/
- private String badRecordsLoggerRedirect;
+ private String badRecordsLoggerAction;
--- End diff --
yes corrected
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---