You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by geetikagupta16 <gi...@git.apache.org> on 2018/02/28 10:53:19 UTC
[GitHub] carbondata pull request #2014: [CARBONDATA-2198] Fixed bug for streaming dat...
GitHub user geetikagupta16 opened a pull request:
https://github.com/apache/carbondata/pull/2014
[CARBONDATA-2198] Fixed bug for streaming data for bad_records_action as REDIRECT or IGNORE
1. Refactored streaming functionality for bad_records_action as IGNORE or REDIRECT
2. Added related test cases
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/geetikagupta16/incubator-carbondata CARBONDATA-2198
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2014.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2014
----
commit ff972f9d32a4eb9786cb9e92c48b574789ae368b
Author: Geetika Gupta <ge...@...>
Date: 2018-02-28T10:39:48Z
1. Refactored streaming functionality for bad_records_action as IGNORE or REDIRECT
2. Added related test cases
----
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/2014
@mohammadshahidkhan Please review. This issue is related to PR #1942
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6007/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5172/
---
[GitHub] carbondata pull request #2014: [CARBONDATA-2198] Fixed bug for streaming dat...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2014#discussion_r171463779
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/streaming/CarbonStreamRecordWriter.java ---
@@ -181,75 +181,81 @@ private void initializeAtFirstRow() throws IOException, InterruptedException {
initializeAtFirstRow();
}
- // parse and convert row
- currentRow.setData(rowParser.parseRow((Object[]) value));
- converter.convert(currentRow);
-
// null bit set
nullBitSet.clear();
- for (int i = 0; i < dataFields.length; i++) {
- if (null == currentRow.getObject(i)) {
- nullBitSet.set(i);
+ Object[] rowData = (Object[]) value;
+ currentRow.setRawData(rowData);
+ // parse and convert row
+ currentRow.setData(rowParser.parseRow(rowData));
+ CarbonRow updatedCarbonRow = converter.convert(currentRow);
+ if (updatedCarbonRow == null) {
+ output.skipRow();
+ currentRow.clearData();
+ } else {
--- End diff --
@geetikagupta16 Thanks for working on this
Hope no change in the else part, only formatting changed due the if check.
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3031/
---
[GitHub] carbondata pull request #2014: [CARBONDATA-2198] Fixed bug for streaming dat...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2014
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2014
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3737/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4848/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2014
LGTM
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2014
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4456/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3981/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4275/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3962/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/2014
LGTM
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3515/
---
[GitHub] carbondata pull request #2014: [CARBONDATA-2198] Fixed bug for streaming dat...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2014#discussion_r171464754
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/streaming/CarbonStreamRecordWriter.java ---
@@ -181,75 +181,81 @@ private void initializeAtFirstRow() throws IOException, InterruptedException {
initializeAtFirstRow();
}
- // parse and convert row
- currentRow.setData(rowParser.parseRow((Object[]) value));
- converter.convert(currentRow);
-
// null bit set
nullBitSet.clear();
- for (int i = 0; i < dataFields.length; i++) {
- if (null == currentRow.getObject(i)) {
- nullBitSet.set(i);
+ Object[] rowData = (Object[]) value;
+ currentRow.setRawData(rowData);
+ // parse and convert row
+ currentRow.setData(rowParser.parseRow(rowData));
+ CarbonRow updatedCarbonRow = converter.convert(currentRow);
+ if (updatedCarbonRow == null) {
+ output.skipRow();
+ currentRow.clearData();
+ } else {
--- End diff --
@mohammadshahidkhan Yes, else part remains the same
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2014
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5017/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2014
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4249/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2736/
---
[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2014
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4742/
---