You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "davisusanibar (via GitHub)" <gi...@apache.org> on 2023/03/21 23:09:24 UTC
[GitHub] [arrow-testing] davisusanibar opened a new pull request, #89: feat: data to test java substrait consumer
davisusanibar opened a new pull request, #89:
URL: https://github.com/apache/arrow-testing/pull/89
Related to https://github.com/apache/arrow/pull/34227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] davisusanibar commented on pull request #89: feat: data to test java substrait consumer
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1487436650
Hi Team,
Just close this PR. Let's start Java Substrait Consumer on the same way that Python/C++ Substrait plans were implemented (maintain the json plan on testing files).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] lidavidm commented on pull request #89: feat: data to test java substrait consumer
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1481204273
Hmm, ok. Depending on Isthmus is a problem for Java since it requires a newer Java version than what we use, so we have to start doing acrobatics with the test setup.
@davisusanibar would you like to start a mailing list discussion about requiring JDK11+ for development & testing in general, but building JARs targeting JDK8? Then we could depend on Isthmus and avoid embedding potentially unstable plans. (Currently, our docs state we can still build on JDK8.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] westonpace commented on pull request #89: feat: data to test java substrait consumer
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1481178615
> @westonpace would it be useful to have more Acero/Substrait testing data like this in apache/arrow?
We have a lot of hard-coded JSON but its embedded in the test files themselves (e.g. serde_test.cc or test_substrait.py) and not in standalone files. The original concern around hard-coded JSON was that Substrait may evolve quickly and those JSON files would be difficult to maintain. For example, the JSON files in this PR are missing the version field (Isthmus does not yet populate this) and they don't have URIs for the extension functions (almost no one generates these yet). So they may need to change at some point.
As a result, I have been waiting for the text format to be ready before I made any attempt to curate a large set of test queries (but that is still a few months off at least).
I think SQL is probably a pretty good solution if you have a good SQL->Substrait library (that may be an advantage for Java). In that case I would suggest only storing the SQL and then generating the Substrait on the fly.
I don't actually know what the legal ramifications are for TPC-H but it is a good question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] westonpace commented on pull request #89: feat: data to test java substrait consumer
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1483005640
> Though to start with I'm not sure if it even needs to be automated per se.
Agreed. As long as the source SQL files are stored alongside the Substrait files, with a detailed README explaining how to regenerate the Substrait files, then I think we can proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] westonpace commented on pull request #89: feat: data to test java substrait consumer
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1481270587
Another option is maybe that we create some kind of docker / ci script in this repo that creates the proto / json files from SQL files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] kou commented on pull request #89: feat: data to test java substrait consumer
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1478796794
Could you remove `.DS_Store`s?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] lidavidm commented on pull request #89: feat: data to test java substrait consumer
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1483042421
OK, then I think what needs to happen here is just check in a Java file (maybe something that'll work with jshell?) showing how to generate the plans given the SQL files, and address the existing review feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] lidavidm commented on pull request #89: feat: data to test java substrait consumer
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #89:
URL: https://github.com/apache/arrow-testing/pull/89#issuecomment-1481293475
Ah, that would work, too. Though to start with I'm not sure if it even needs to be automated per se.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] davisusanibar closed pull request #89: feat: data to test java substrait consumer
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar closed pull request #89: feat: data to test java substrait consumer
URL: https://github.com/apache/arrow-testing/pull/89
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org