You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by mohammadshahidkhan <gi...@git.apache.org> on 2018/07/17 11:38:09 UTC
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
GitHub user mohammadshahidkhan opened a pull request:
https://github.com/apache/carbondata/pull/2517
[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full.
…
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed?
NA
- [X] Any backward compatibility impacted?
NA
- [X] Document update required?
NA
- [X] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
NA
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata tablestatus_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2517.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2517
----
commit 46b253e04d8ed84798db1abae6e64f7b62c4523f
Author: mohammadshahidkhan <mo...@...>
Date: 2018-07-17T11:29:35Z
[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full.
----
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on the issue:
https://github.com/apache/carbondata/pull/2517
LGTM
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5955/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5966/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7572/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6025/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest SDV please
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6113/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest sdv please
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7349/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6207/
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r205748318
--- Diff: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java ---
@@ -724,6 +724,7 @@ public void writeLoadDetailsIntoFile(List<SegmentUpdateDetails> listOfSegmentUpd
brWriter.write(metadataInstance);
} catch (IOException ioe) {
LOG.error("Error message: " + ioe.getLocalizedMessage());
+ fileWrite.setFailed();
--- End diff --
At some places IO exception is thrown back after setting the flag and some places exception is not thrown..any specific reason for this?...can we keep the behavior same..better to throw back the exception at all places
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/2517
The failled test case is unrelated and random.
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5903/
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r203957912
--- Diff: core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java ---
@@ -70,12 +78,20 @@ public AtomicFileOperationsImpl(String filePath, FileType fileType) {
if (null != dataOutStream) {
CarbonUtil.closeStream(dataOutStream);
CarbonFile tempFile = FileFactory.getCarbonFile(tempWriteFilePath, fileType);
- if (!tempFile.renameForce(filePath)) {
- throw new IOException("temporary file renaming failed, src="
- + tempFile.getPath() + ", dest=" + filePath);
+ if (!this.setFailed) {
+ if (!tempFile.renameForce(filePath)) {
+ throw new IOException(
+ "temporary file renaming failed, src=" + tempFile.getPath() + ", dest=" + filePath);
+ }
}
+ } else {
+ LOGGER.warn("The temporary file renaming skipped due to I/O error, deleting file "
+ + tempWriteFilePath);
}
}
+ @Override public void setFailed() {
--- End diff --
Fixed
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2517
LGTM. The failures in 2.2.1 build are note related to this PR. Those are failing on master as well. Need to fix.
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7450/
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r204685081
--- Diff: core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java ---
@@ -70,12 +78,20 @@ public AtomicFileOperationsImpl(String filePath, FileType fileType) {
if (null != dataOutStream) {
CarbonUtil.closeStream(dataOutStream);
CarbonFile tempFile = FileFactory.getCarbonFile(tempWriteFilePath, fileType);
- if (!tempFile.renameForce(filePath)) {
- throw new IOException("temporary file renaming failed, src="
- + tempFile.getPath() + ", dest=" + filePath);
+ if (!this.setFailed) {
+ if (!tempFile.renameForce(filePath)) {
+ throw new IOException(
+ "temporary file renaming failed, src=" + tempFile.getPath() + ", dest=" + filePath);
+ }
}
+ } else {
+ LOGGER.warn("The temporary file renaming skipped due to I/O error, deleting file "
--- End diff --
@KanakaKumar Thanks for the review.
Actually when the delete fails the temp file will persists and its better to delete it at the same time, to avoid problem in secure environment.
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest sdv please
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r204456375
--- Diff: core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java ---
@@ -70,12 +78,20 @@ public AtomicFileOperationsImpl(String filePath, FileType fileType) {
if (null != dataOutStream) {
CarbonUtil.closeStream(dataOutStream);
CarbonFile tempFile = FileFactory.getCarbonFile(tempWriteFilePath, fileType);
- if (!tempFile.renameForce(filePath)) {
- throw new IOException("temporary file renaming failed, src="
- + tempFile.getPath() + ", dest=" + filePath);
+ if (!this.setFailed) {
+ if (!tempFile.renameForce(filePath)) {
+ throw new IOException(
+ "temporary file renaming failed, src=" + tempFile.getPath() + ", dest=" + filePath);
+ }
}
+ } else {
+ LOGGER.warn("The temporary file renaming skipped due to I/O error, deleting file "
--- End diff --
Here actually delete code is added. AtomicFileOperationsImpl is taking care of overwriting temp file even if it exists. So, please correct the message accordingly.
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7489/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest sdv please
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6001/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6038/
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2517
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5982/
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r203411199
--- Diff: core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java ---
@@ -70,12 +78,20 @@ public AtomicFileOperationsImpl(String filePath, FileType fileType) {
if (null != dataOutStream) {
CarbonUtil.closeStream(dataOutStream);
CarbonFile tempFile = FileFactory.getCarbonFile(tempWriteFilePath, fileType);
- if (!tempFile.renameForce(filePath)) {
- throw new IOException("temporary file renaming failed, src="
- + tempFile.getPath() + ", dest=" + filePath);
+ if (!this.setFailed) {
+ if (!tempFile.renameForce(filePath)) {
+ throw new IOException(
+ "temporary file renaming failed, src=" + tempFile.getPath() + ", dest=" + filePath);
+ }
}
+ } else {
+ LOGGER.warn("The temporary file renaming skipped due to I/O error, deleting file "
+ + tempWriteFilePath);
}
}
+ @Override public void setFailed() {
--- End diff --
setFailed needs to be raised on Exception cases in the below usages of AtomicFileOperations
- org.apache.carbondata.core.datamap.status.DiskBasedDataMapStatusProvider#writeLoadDetailsIntoFile
- org.apache.carbondata.core.metadata.SegmentFileStore#writeSegmentFile(org.apache.carbondata.core.metadata.SegmentFileStore.SegmentFile, java.lang.String)
- org.apache.carbondata.core.statusmanager.SegmentStatusManager#writeLoadMetadata
Please also handle any other references I missed.
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r204702468
--- Diff: core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java ---
@@ -70,12 +78,20 @@ public AtomicFileOperationsImpl(String filePath, FileType fileType) {
if (null != dataOutStream) {
CarbonUtil.closeStream(dataOutStream);
CarbonFile tempFile = FileFactory.getCarbonFile(tempWriteFilePath, fileType);
- if (!tempFile.renameForce(filePath)) {
- throw new IOException("temporary file renaming failed, src="
- + tempFile.getPath() + ", dest=" + filePath);
+ if (!this.setFailed) {
+ if (!tempFile.renameForce(filePath)) {
+ throw new IOException(
+ "temporary file renaming failed, src=" + tempFile.getPath() + ", dest=" + filePath);
+ }
}
+ } else {
+ LOGGER.warn("The temporary file renaming skipped due to I/O error, deleting file "
--- End diff --
Ok fine.
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7260/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6012/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6330/
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r205752516
--- Diff: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java ---
@@ -724,6 +724,7 @@ public void writeLoadDetailsIntoFile(List<SegmentUpdateDetails> listOfSegmentUpd
brWriter.write(metadataInstance);
} catch (IOException ioe) {
LOG.error("Error message: " + ioe.getLocalizedMessage());
+ fileWrite.setFailed();
--- End diff --
@manishgupta88 Thanks for the review.
Agreed with your point eating exception is not a good idea, handled throwing back at all the places.
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest sdv please
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7504/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest sdv please
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest sdv please
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2517
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5903/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6247/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/2517
retest sdv please
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6261/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7499/
---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2517
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6024/
---