You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/03 01:51:10 UTC
[GitHub] [hudi] XuQianJin-Stars opened a new pull request, #6284: improve spillableMapBasePath disk directory is full
XuQianJin-Stars opened a new pull request, #6284:
URL: https://github.com/apache/hudi/pull/6284
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1294401196
> let me explain my understanding. by default, we create spillable map dir files using `deleteOnExit`. So, only case is, if process crashed mid-way, those files are left as is.
>
> So, irrespective of whether we chose "/tmp" or some other dir "X" chosen by user. there should be some other process that cleans up either /tmp" or "X" at regular cadence. if not, not sure how this would automatically solve the issue. thats the only way the spillable base path could get full.
>
> or am I missing something here.
This directory in yarn is now deleted when the app exits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1304439073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9aae324aa3fcf951f4fcfcff5d88555cf087b61 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791)
* ced81ffe44bdb817d55ab972fff0517975931fb2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r937334947
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> So why the dir is full if it is cleaned in time ?
When spark writes multiple jobs concurrently, there are not only hudi jobs, but the `tmp` directory shared by many jobs will cause it to explode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1278738890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 2c5730beb0901835dbea259823620d7ebc4dbecc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1254371638
@xushiyan : can you review this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1283288205
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
}, {
"hash" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 7d79e887c7ae4027c7351547a2224cd2693ea35f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282)
* ff2defa78be7cda445942ed897dd7e6e049ed6eb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1306545134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
}, {
"hash" : "1d4477334ee7d158111a01559641015379597fc0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827",
"triggerID" : "1d4477334ee7d158111a01559641015379597fc0",
"triggerType" : "PUSH"
}, {
"hash" : "1d4477334ee7d158111a01559641015379597fc0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12864",
"triggerID" : "1306543690",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1d4477334ee7d158111a01559641015379597fc0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12864)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1278464269
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 0946f43b44aae6a01ecc6a610d9bd48404aaaacd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154)
* 2c5730beb0901835dbea259823620d7ebc4dbecc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994232758
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
Review Comment:
> Thanks, should we take the write config as higher priority then ? If user configured it explicitly.
It may be necessary to remove the default value
```
public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
.key("hoodie.memory.spillable.map.path")
.defaultValue("/tmp/")
.withDocumentation("Default file path prefix for spillable map");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r936208300
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> So when we clean these files then ?
1.`BitCaskDiskMap::initFile` set `writeOnlyFile.deleteOnExit()`
2. `RocksDbDiskMap::close`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1272437525
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 82dd925f9018c0ec3fb3bfaa09f70174010af90c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079)
* a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1305219205
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
}, {
"hash" : "1d4477334ee7d158111a01559641015379597fc0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827",
"triggerID" : "1d4477334ee7d158111a01559641015379597fc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ced81ffe44bdb817d55ab972fff0517975931fb2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799)
* 1d4477334ee7d158111a01559641015379597fc0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1204738947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 0ea0766862c16ccec08c7c621f98ca8402f772ff Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1254378397
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 0ea0766862c16ccec08c7c621f98ca8402f772ff Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571)
* 4b0a4e72766491e15dbeb8ed904c9aabae32bb89 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994243300
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
Review Comment:
> Thanks, should we take the write config as higher priority then ? If user configured it explicitly.
It may be necessary to remove the default value
```
public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
.key("hoodie.memory.spillable.map.path")
.defaultValue("/tmp/")
.withDocumentation("Default file path prefix for spillable map");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1272425889
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 4b0a4e72766491e15dbeb8ed904c9aabae32bb89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563)
* 82dd925f9018c0ec3fb3bfaa09f70174010af90c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1282092984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 7d79e887c7ae4027c7351547a2224cd2693ea35f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1306752083
hi @nsivabalan i adressed this minor comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1203601806
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* d8d834d4f66fe195d009331395c12f3302aaaa9e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1275854076
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081)
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1275936356
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081)
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 0946f43b44aae6a01ecc6a610d9bd48404aaaacd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994473005
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -93,9 +94,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
instantRange, withOperationField,
forceFullScan, partitionName, internalSchema);
try {
+ String[] localDirs = FileIOUtils.getConfiguredLocalDirs();
+ spillableMapBasePath = localDirs.length > 0 ? localDirs[0] : spillableMapBasePath;
Review Comment:
this is to silently overwrite user-input config, which we should avoid. We should always respect user-provided setting, then fallback to infer from environment, then fallback to a sane default like the existing `/tmp/`. I'd suggest improve the `SPILLABLE_MAP_BASE_PATH` with inferring function.
```java
public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
.key("hoodie.memory.spillable.map.path")
.defaultValue("/tmp/")
.withInferFunction(() -> {
// if user doesn't configure it, infer the path from env var
// if nothing to infer from, return Option.empty() then it will fallback to the default /tmp/
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1303571683
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
}, {
"hash" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320",
"triggerID" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"triggerType" : "PUSH"
}, {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* ff2defa78be7cda445942ed897dd7e6e049ed6eb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320)
* d9aae324aa3fcf951f4fcfcff5d88555cf087b61 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1272455607
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1305213429
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
}, {
"hash" : "1d4477334ee7d158111a01559641015379597fc0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1d4477334ee7d158111a01559641015379597fc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ced81ffe44bdb817d55ab972fff0517975931fb2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799)
* 1d4477334ee7d158111a01559641015379597fc0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1304439780
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9aae324aa3fcf951f4fcfcff5d88555cf087b61 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791)
* ced81ffe44bdb817d55ab972fff0517975931fb2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1203465179
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 416a0a367b5490312b4584dd543710e53766bc11 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r938371723
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
not sure if we can do this. spillableMapbase path is configurable. If one does not want "/tmp/" which is the default, they can always override using the configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994153351
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
Review Comment:
> We totally ignore the option through hoodie write config then ?
First, get the local dir from the yarn container. If you can't get it, you can get it from the hoodie write config. In this way, users on yarn basically do not need to actively modify the hoodie write config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1275942532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081)
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 0946f43b44aae6a01ecc6a610d9bd48404aaaacd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r990721773
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
basepath is hdfs path, spillableMapbase_path is local path,
It is wrong to use hdfs path directly as spillableMapbase_path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1272436538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 4b0a4e72766491e15dbeb8ed904c9aabae32bb89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563)
* 82dd925f9018c0ec3fb3bfaa09f70174010af90c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079)
* a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1204708180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* d8d834d4f66fe195d009331395c12f3302aaaa9e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544)
* 0ea0766862c16ccec08c7c621f98ca8402f772ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r1014904091
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
+ // to what Yarn on this system said was available. Note this assumes that Yarn has
+ // created the directories already, and that they are secured so that only the
+ // user has access to them.
+ return getYarnLocalDirs().split(",");
+ } else if (System.getProperty("java.io.tmpdir") != null) {
+ return System.getProperty("java.io.tmpdir").split(",");
+ } else {
+ return null;
+ }
+ }
+
+ public static boolean isRunningInYarnContainer() {
+ // These environment variables are set by YARN.
+ return System.getenv("CONTAINER_ID") != null;
+ }
+
+ /**
+ * Get the Yarn approved local directories.
+ */
+ public static String getYarnLocalDirs() {
Review Comment:
same comment as above.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
+ // to what Yarn on this system said was available. Note this assumes that Yarn has
+ // created the directories already, and that they are secured so that only the
+ // user has access to them.
+ return getYarnLocalDirs().split(",");
+ } else if (System.getProperty("java.io.tmpdir") != null) {
+ return System.getProperty("java.io.tmpdir").split(",");
+ } else {
+ return null;
+ }
+ }
+
+ public static boolean isRunningInYarnContainer() {
Review Comment:
why these are public? can we make it private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1306763537
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
}, {
"hash" : "1d4477334ee7d158111a01559641015379597fc0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827",
"triggerID" : "1d4477334ee7d158111a01559641015379597fc0",
"triggerType" : "PUSH"
}, {
"hash" : "1d4477334ee7d158111a01559641015379597fc0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12864",
"triggerID" : "1306543690",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1d4477334ee7d158111a01559641015379597fc0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12864)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #6284:
URL: https://github.com/apache/hudi/pull/6284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1303960351
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
}, {
"hash" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320",
"triggerID" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"triggerType" : "PUSH"
}, {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* d9aae324aa3fcf951f4fcfcff5d88555cf087b61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1304413329
CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1203550778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 416a0a367b5490312b4584dd543710e53766bc11 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534)
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* d8d834d4f66fe195d009331395c12f3302aaaa9e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1281842830
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 2c5730beb0901835dbea259823620d7ebc4dbecc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200)
* 7d79e887c7ae4027c7351547a2224cd2693ea35f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1283293621
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
}, {
"hash" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320",
"triggerID" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 7d79e887c7ae4027c7351547a2224cd2693ea35f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282)
* ff2defa78be7cda445942ed897dd7e6e049ed6eb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1278467230
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 0946f43b44aae6a01ecc6a610d9bd48404aaaacd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154)
* 2c5730beb0901835dbea259823620d7ebc4dbecc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994137729
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
Review Comment:
We totally ignore the option through hoodie write config then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r993215377
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,46 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
+ // to what Yarn on this system said was available. Note this assumes that Yarn has
+ // created the directories already, and that they are secured so that only the
+ // user has access to them.
+ return getYarnLocalDirs().split(",");
+ } else if (System.getenv("SPARK_EXECUTOR_DIRS") != null) {
+ return System.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator);
+ } else if (System.getenv("SPARK_LOCAL_DIRS") != null) {
+ return System.getenv("SPARK_LOCAL_DIRS").split(",");
+ } else {
+ // In non-Yarn mode (or for the driver in yarn-client mode), we cannot trust the user
+ // configuration to point to a secure directory. So create a subdirectory with restricted
+ // permissions under each listed directory.
+ return Option.of(System.getProperty("spark.local.dir"))
+ .orElse(System.getProperty("java.io.tmpdir")).split(",");
+ }
+ }
+
+ public static Boolean isRunningInYarnContainer() {
Review Comment:
Boolean -> boolean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1276122772
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 0946f43b44aae6a01ecc6a610d9bd48404aaaacd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1304414759
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9aae324aa3fcf951f4fcfcff5d88555cf087b61 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1304413781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9aae324aa3fcf951f4fcfcff5d88555cf087b61 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r937313484
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
So why the dir is full if it is cleaned in time ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r937334947
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> So why the dir is full if it is cleaned in time ?
When spark writes multiple jobs concurrently, there are not only hudi jobs, but the tmp directory shared by many jobs will cause it to explode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1254469770
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 4b0a4e72766491e15dbeb8ed904c9aabae32bb89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1275856728
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994162105
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
Review Comment:
> If you can't get it, you can get it from the hoodie write config
I didn't see the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994179170
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
Review Comment:
> > If you can't get it, you can get it from the hoodie write config
>
> I didn't see the logic.
https://github.com/apache/hudi/blob/0946f43b44aae6a01ecc6a610d9bd48404aaaacd/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java#L98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1203403145
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 416a0a367b5490312b4584dd543710e53766bc11 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r936208300
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> So when we clean these files then ?
1. `BitCaskDiskMap` set `writeOnlyFile.deleteOnExit()`
2. `RocksDbDiskMap::close`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r936208300
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> So when we clean these files then ?
`BitCaskDiskMap::initFile` set `writeOnlyFile.deleteOnExit()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1203547322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 416a0a367b5490312b4584dd543710e53766bc11 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534)
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6284: improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r936189759
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
So when we clean these files then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r938382783
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> not sure if we can do this. spillableMapbase path is configurable. If one does not want "/tmp/" which is the default, they can always override using the configs.
It is more troublesome for users to use, and this option needs to be specified additionally, and considering that the `basepath` will definitely be in a large data disk, the temporary directory can be put into the same disk as the `basepath`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r938382783
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> not sure if we can do this. spillableMapbase path is configurable. If one does not want "/tmp/" which is the default, they can always override using the configs.
It is more troublesome for users to use, and this option needs to be specified additionally, and considering that the basepath will definitely be in a large data disk, the temporary directory can be put into the same disk as the basepath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r1000140401
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java:
##########
@@ -77,10 +78,20 @@ public class HoodieMemoryConfig extends HoodieConfig {
.defaultValue(16 * 1024 * 1024)
.withDocumentation("Property to control the max memory in bytes for dfs input stream buffer size");
+ public static final String MEMORY_SPILLABLE_MAP_BASE_PATH = "hoodie.memory.spillable.map.path";
public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
- .key("hoodie.memory.spillable.map.path")
+ .key(MEMORY_SPILLABLE_MAP_BASE_PATH)
.defaultValue("/tmp/")
- .withDocumentation("Default file path prefix for spillable map");
+ .withInferFunction(cfg -> {
+ // if user doesn't configure it, infer the path from env var
+ // if nothing to infer from, return Option.empty() then it will fallback to the default /tmp/
+ if (cfg.contains(MEMORY_SPILLABLE_MAP_BASE_PATH)) {
+ return Option.of(cfg.getString(MEMORY_SPILLABLE_MAP_BASE_PATH));
Review Comment:
this check is not needed. infer function is invoked only when user does not set it
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java:
##########
@@ -77,10 +78,20 @@ public class HoodieMemoryConfig extends HoodieConfig {
.defaultValue(16 * 1024 * 1024)
.withDocumentation("Property to control the max memory in bytes for dfs input stream buffer size");
+ public static final String MEMORY_SPILLABLE_MAP_BASE_PATH = "hoodie.memory.spillable.map.path";
public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
- .key("hoodie.memory.spillable.map.path")
+ .key(MEMORY_SPILLABLE_MAP_BASE_PATH)
Review Comment:
no need to extract to constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1303461440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
}, {
"hash" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320",
"triggerID" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"triggerType" : "PUSH"
}, {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* ff2defa78be7cda445942ed897dd7e6e049ed6eb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320)
* d9aae324aa3fcf951f4fcfcff5d88555cf087b61 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1252945374
@XuQianJin-Stars : will you have time to address the feedback in the next few days. we have code freeze coming up for 0.12.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1272426439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 4b0a4e72766491e15dbeb8ed904c9aabae32bb89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563)
* 82dd925f9018c0ec3fb3bfaa09f70174010af90c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r994217722
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java:
##########
@@ -204,4 +204,40 @@ public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.
public static Option<byte[]> readDataFromPath(FileSystem fileSystem, org.apache.hadoop.fs.Path detailPath) {
return readDataFromPath(fileSystem, detailPath, false);
}
+
+ /**
+ * Return the configured local directories where hudi can write files. This
+ * method does not create any directories on its own, it only encapsulates the
+ * logic of locating the local directories according to deployment mode.
+ */
+ public static String[] getConfiguredLocalDirs() {
+ if (isRunningInYarnContainer()) {
+ // If we are in yarn mode, systems can have different disk layouts so we must set it
Review Comment:
Thanks, should we take the write config as higher priority then ? If user configured it explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1204705699
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* d8d834d4f66fe195d009331395c12f3302aaaa9e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544)
* 0ea0766862c16ccec08c7c621f98ca8402f772ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1203554590
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 416a0a367b5490312b4584dd543710e53766bc11 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534)
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* d8d834d4f66fe195d009331395c12f3302aaaa9e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1203400609
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 416a0a367b5490312b4584dd543710e53766bc11 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #6284: improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r936208300
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -92,11 +92,12 @@ protected HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<Stri
forceFullScan, partitionName, internalSchema);
try {
// Store merged records for all versions for this log file, set the in-memory footprint to maxInMemoryMapSize
- this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
+ this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, basePath + spillableMapBasePath, new DefaultSizeEstimator(),
new HoodieRecordSizeEstimator(readerSchema), diskMapType, isBitCaskDiskMapCompressionEnabled);
+
Review Comment:
> So when we clean these files then ?
1. `BitCaskDiskMap::initFile` set `writeOnlyFile.deleteOnExit()`
2. `RocksDbDiskMap::close`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1248761759
I don't want to base the spillable base path w/ the table base path by default.
may be, you can introduce a config called "hoodie.memory.use.table.path.for.spillable.map". and we can add .hoodie/spillablemap/ and start using that if this config is enabled. if not, we will fallback to the value set for `hoodie.memory.spillable.map.path`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1306543690
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1304437063
> asked for some clarification
> asked for some clarification
hi @nsivabalan do you have some questions about this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1304496321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ced81ffe44bdb817d55ab972fff0517975931fb2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1305424690
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12791",
"triggerID" : "d9aae324aa3fcf951f4fcfcff5d88555cf087b61",
"triggerType" : "PUSH"
}, {
"hash" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12799",
"triggerID" : "ced81ffe44bdb817d55ab972fff0517975931fb2",
"triggerType" : "PUSH"
}, {
"hash" : "1d4477334ee7d158111a01559641015379597fc0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827",
"triggerID" : "1d4477334ee7d158111a01559641015379597fc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1d4477334ee7d158111a01559641015379597fc0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12827)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6284:
URL: https://github.com/apache/hudi/pull/6284#discussion_r996967389
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -219,6 +219,11 @@ public class HoodieTableConfig extends HoodieConfig {
.defaultValue(false)
.withDocumentation("When set to true, will not write the partition columns into hudi. By default, false.");
+ public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
+ .key("hoodie.memory.spillable.map.path")
Review Comment:
shouldn't this be `hoodie.memory.spillable.map.basepath` ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java:
##########
@@ -77,10 +80,19 @@ public class HoodieMemoryConfig extends HoodieConfig {
.defaultValue(16 * 1024 * 1024)
.withDocumentation("Property to control the max memory in bytes for dfs input stream buffer size");
- public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
+ public static ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
Review Comment:
why remove `final` ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java:
##########
@@ -77,10 +80,19 @@ public class HoodieMemoryConfig extends HoodieConfig {
.defaultValue(16 * 1024 * 1024)
.withDocumentation("Property to control the max memory in bytes for dfs input stream buffer size");
- public static final ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
+ public static ConfigProperty<String> SPILLABLE_MAP_BASE_PATH = ConfigProperty
.key("hoodie.memory.spillable.map.path")
.defaultValue("/tmp/")
- .withDocumentation("Default file path prefix for spillable map");
+ .withInferFunction(cfg -> {
+ // if user doesn't configure it, infer the path from env var
+ // if nothing to infer from, return Option.empty() then it will fallback to the default /tmp/
+ if (cfg.contains(HoodieTableConfig.SPILLABLE_MAP_BASE_PATH)) {
+ return Option.of(cfg.getString(HoodieTableConfig.SPILLABLE_MAP_BASE_PATH));
+ }
+ String[] localDirs = FileIOUtils.getConfiguredLocalDirs();
+ return (localDirs != null && localDirs.length > 0) ? Option.of(localDirs[0]) : Option.of("/tmp/");
Review Comment:
/nit return option.empty means using the default val
```suggestion
return (localDirs != null && localDirs.length > 0) ? Option.of(localDirs[0]) : Option.empty();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1254375965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* 0ea0766862c16ccec08c7c621f98ca8402f772ff Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571)
* 4b0a4e72766491e15dbeb8ed904c9aabae32bb89 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1281839655
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* 2c5730beb0901835dbea259823620d7ebc4dbecc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200)
* 7d79e887c7ae4027c7351547a2224cd2693ea35f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6284: [HUDI-4526] Improve spillableMapBasePath disk directory is full
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6284:
URL: https://github.com/apache/hudi/pull/6284#issuecomment-1283469357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "416a0a367b5490312b4584dd543710e53766bc11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10534",
"triggerID" : "416a0a367b5490312b4584dd543710e53766bc11",
"triggerType" : "PUSH"
}, {
"hash" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026dbfc7a6d4d7e489e8c8671a84e143bdb01758",
"triggerType" : "PUSH"
}, {
"hash" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10544",
"triggerID" : "d8d834d4f66fe195d009331395c12f3302aaaa9e",
"triggerType" : "PUSH"
}, {
"hash" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10571",
"triggerID" : "0ea0766862c16ccec08c7c621f98ca8402f772ff",
"triggerType" : "PUSH"
}, {
"hash" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11563",
"triggerID" : "4b0a4e72766491e15dbeb8ed904c9aabae32bb89",
"triggerType" : "PUSH"
}, {
"hash" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12079",
"triggerID" : "82dd925f9018c0ec3fb3bfaa09f70174010af90c",
"triggerType" : "PUSH"
}, {
"hash" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12081",
"triggerID" : "a455e4c67d1ac237ef999ac8d6aa584af2f4cd1f",
"triggerType" : "PUSH"
}, {
"hash" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed6ba26ec0c2a05f36eb121d404742fb948caf4b",
"triggerType" : "PUSH"
}, {
"hash" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12154",
"triggerID" : "0946f43b44aae6a01ecc6a610d9bd48404aaaacd",
"triggerType" : "PUSH"
}, {
"hash" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12200",
"triggerID" : "2c5730beb0901835dbea259823620d7ebc4dbecc",
"triggerType" : "PUSH"
}, {
"hash" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12282",
"triggerID" : "7d79e887c7ae4027c7351547a2224cd2693ea35f",
"triggerType" : "PUSH"
}, {
"hash" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320",
"triggerID" : "ff2defa78be7cda445942ed897dd7e6e049ed6eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026dbfc7a6d4d7e489e8c8671a84e143bdb01758 UNKNOWN
* ed6ba26ec0c2a05f36eb121d404742fb948caf4b UNKNOWN
* ff2defa78be7cda445942ed897dd7e6e049ed6eb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org