You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "zhfeng (via GitHub)" <gi...@apache.org> on 2023/07/03 01:04:57 UTC
[GitHub] [camel-quarkus] zhfeng opened a new pull request, #5041: Re-enable google-bigquery and add a work around
zhfeng opened a new pull request, #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041
<!-- Uncomment and fill this section if your PR is not trivial
[ ] An issue should be filed for the change unless this is a trivial change (fixing a typo or similar). One issue should ideally be fixed by not more than one commit and the other way round, each commit should fix just one issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful and properly spelled subject line and body. Copying the title of the associated issue is typically enough. Please include the issue number in the commit message prefixed by #.
[ ] The pull request description should explain what the pull request does, how, and why. If the info is available in the associated issue or some other external document, a link is enough.
[ ] Phrases like Fix #<issueNumber> or Fixes #<issueNumber> will auto-close the named issue upon merging the pull request. Using them is typically a good idea.
[ ] Please run mvn process-resources -Pformat (and amend the changes if necessary) before sending the pull request.
[ ] Contributor guide is your good friend: https://camel.apache.org/camel-quarkus/latest/contributor-guide.html
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] zhfeng commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "zhfeng (via GitHub)" <gi...@apache.org>.
zhfeng commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1619433270
Sine `org.apache.arrow:arrow-memory-netty` is a runtime dependency, `RemovedResourceBuildItem` seems not working. I just try to copy 4 files from `apache-arrow`
- LargeBuffer.java
- MutableWrappedByteBuf.java
- PooledByteBufAllocatorL.java
- UnsafeDirectLittleEndian.java
and apply the fix for `PooledByteBufAllocatorL.java`. All of the `google-bigquery` tests look good both in JVM and native mode.
Also I think we need a test for a large table and enable the result with arrow data format. Can it be supported by wiremock?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] zhfeng commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "zhfeng (via GitHub)" <gi...@apache.org>.
zhfeng commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1618629795
@jamesnetherton I think `apache-arrow` does not shade Netty. The problem is Quarkus 3.2.0.Final update Netty to `4.1.94.Final` and there is an un-compatible change in `PooledByteBufAllocator`. So I think it could also impact the JVM mode if we test with a large table.
The fix in `apache-arrow` is https://github.com/apache/arrow/commit/ea4f03ac166e3961b59f29f60dcd160fbed29894#diff-d967dd684919ce097b81e439fafc96217f1263985f5864faa4b78eb96e7a4f16L164
So is it possible to have this file in our `camel-quarkus-google-bigquery`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1619557554
> Can it be supported by wiremock?
Yes, probably. Not sure if it'd mean storing huge files for the stubbed data in the repo though.....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1618723170
Ah, yeah. Sorry for the confusion!
> So is it possible to have this file in our camel-quarkus-google-bigquery?
We could temporarily have it in our extension. Quarkus has `RemovedResourceBuildItem`, not sure if that could also be used to remove the bad impl from Arrow?
If it becomes too much work. Maybe we just stick with the original plan of removing Arrow until we get the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1617426190
We should add the same doc notes for the workaround like we did for pubsub & storage.
Also, not sure how safe the Arrow exclusion is. It is referenced in a few places:
https://github.com/search?q=repo%3Agoogleapis%2Fjava-bigquery+arrow+language%3AJava&type=code&l=Java
I guess the fact that the native compilation + tests passed means that maybe we don't hit those code paths in Camel...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] zhfeng commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "zhfeng (via GitHub)" <gi...@apache.org>.
zhfeng commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1619632162
I raise
- https://github.com/apache/camel-quarkus/issues/5048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1618280972
@zhfeng is it just the shaded Netty of Arrow that is causing the problem? If Arrow does not modify any of the Netty bits, maybe we could try something similar to what we did for Kudu and shaded Netty? See here: https://github.com/apache/camel-quarkus/pull/2008 & https://github.com/apache/camel-quarkus/issues/1907.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] zhfeng commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "zhfeng (via GitHub)" <gi...@apache.org>.
zhfeng commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1617534563
Yeah, it is exactly what I was concerned. It seems that `google-bigquery` default uses `DataFormat.ARROW` when read/write a large data table result.
We don't hit those codes because we lack of a test for the large table. So how can write a sentence for this limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] zhfeng merged pull request #5041: Re-enable google-bigquery and add a work around
Posted by "zhfeng (via GitHub)" <gi...@apache.org>.
zhfeng merged PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "ppalaga (via GitHub)" <gi...@apache.org>.
ppalaga commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1619584991
> storing huge files for the stubbed data
We could perhaps generate those at build time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5041: Re-enable google-bigquery and add a work around
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5041:
URL: https://github.com/apache/camel-quarkus/pull/5041#issuecomment-1618254304
Thinking a bit more, I'm not sure I like excluding Arrow because it potentially impacts negatively on JVM mode, and the issue is purely a native mode one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org