You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "dmarsh19 (via GitHub)" <gi...@apache.org> on 2023/12/08 21:30:53 UTC
[PR] camel-parquet-avro: Add the capability to lazy load records during unmarshal [camel]
dmarsh19 opened a new pull request, #12385:
URL: https://github.com/apache/camel/pull/12385
# Description
Currently, the parquet-avro data format will read all records at once in to memory. This PR adds an option `lazyLoad` that reads records one by one via an iterator. This avoids some potential memory issues when processing large parquet files.
# Target
- [X] I checked that the commit is targeting the correct branch (note that Camel 3 uses `camel-3.x`, whereas Camel 4 uses the `main` branch)
# Tracking
- [X] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
There is no JIRA issue for this PR, I'm not sure the process to have one created. I'll gladly help get one created and update anything I need to on this PR.
# Apache Camel coding standards and style
- [X] I checked that each commit in the pull request has a meaningful subject line and body.
- [X] I have run `mvn clean install -DskipTests` locally and I have committed all auto-generated changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] camel-parquet-avro: Add the capability to lazy load records during unmarshal [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12385:
URL: https://github.com/apache/camel/pull/12385#issuecomment-1847866399
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot.
* You can label PRs using `build-all`, `build-dependents`, `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the Summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
* :warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] camel-parquet-avro: Add the capability to lazy load records during unmarshal [camel]
Posted by "dmarsh19 (via GitHub)" <gi...@apache.org>.
dmarsh19 commented on PR #12385:
URL: https://github.com/apache/camel/pull/12385#issuecomment-1847867997
See my note on no JIRA issue
> There is no JIRA issue for this PR, I'm not sure the process to have one created. I'll gladly help get one created and update anything I need to on this PR.
Let me know what I can do to help.
I can resolve any conflicts on generated files as needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] camel-parquet-avro: Add the capability to lazy load records during unmarshal [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #12385:
URL: https://github.com/apache/camel/pull/12385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] camel-parquet-avro: Add the capability to lazy load records during unmarshal [camel]
Posted by "dmarsh19 (via GitHub)" <gi...@apache.org>.
dmarsh19 commented on PR #12385:
URL: https://github.com/apache/camel/pull/12385#issuecomment-1847932206
sorry about the git mess and force push
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org