You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "fathollahzadeh (via GitHub)" <gi...@apache.org> on 2023/07/17 13:01:05 UTC
[GitHub] [systemds] fathollahzadeh opened a new pull request, #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
fathollahzadeh opened a new pull request, #1865:
URL: https://github.com/apache/systemds/pull/1865
This is an updated implementation of GIO, a system for generating efficient matrix and frame readers for custom data formats by example. Our GIO paper was published in SIGMOD 2023. The main improvements and extensions over the previous implementation are:
1. Support for both frame and matrix reader generation
2. Support for projection over the selected fields
3. Improved code generation that adapts to single/multi-threaded configuration
4. Support for single/multi-line record representations
Thanks for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] fathollahzadeh commented on pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "fathollahzadeh (via GitHub)" <gi...@apache.org>.
fathollahzadeh commented on PR #1865:
URL: https://github.com/apache/systemds/pull/1865#issuecomment-1638353730
> Also, src/test/scripts/functions/iogen/data/Custom/F7/sample-aminer-author200.raw seems to be copied many times, are there differences in these files?
resolved in the last commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #1865:
URL: https://github.com/apache/systemds/pull/1865#issuecomment-1638261907
Also, src/test/scripts/functions/iogen/data/Custom/F7/sample-aminer-author200.raw
seems to be copied many times, are there differences in these files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard closed pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard closed pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
URL: https://github.com/apache/systemds/pull/1865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #1865:
URL: https://github.com/apache/systemds/pull/1865#issuecomment-1641723366
> > It is just missing an exception on :
> > https://github.com/apache/systemds/blob/715b2fc523f700974ad1aa348d155553ed1bc52f/pom.xml#L636
> >
> > for the path: src/test/scripts/functions/iogen/in/
>
> can I add this line to pom.xml? `<exclude>src/test/scripts/functions/iogen/in/*/*/*</exclude> ` or, should I find another way?
yes, feel free to add it to the pom.xml. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #1865:
URL: https://github.com/apache/systemds/pull/1865#issuecomment-1642728303
Thanks for the contribution @fathollahzadeh ! I have merged it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] fathollahzadeh commented on pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "fathollahzadeh (via GitHub)" <gi...@apache.org>.
fathollahzadeh commented on PR #1865:
URL: https://github.com/apache/systemds/pull/1865#issuecomment-1638352355
> Hi @fathollahzadeh
>
> Thanks for the isolated PR, i scrolled through it and it does not touch any files that would break existing code. I can take a closer look at another time, but given that it looks well tested i say if the automated tests pass then we merge it!
>
> I have one question/request, if we can avoid adding one file : src/test/scripts/functions/iogen/data/XML/sample-autolead-xml200.raw
>
> This is 21400 lines, can the test of xml parsing be reduced?
>
> Best regards Sebastian
Hi @Baunsgaard
Thank you for the prompt feedback and suggestions.
I have made the following changes:
1. Changed the dataset directory from "data" to "in", which should resolve the license error.
2. Removed duplicate datasets
3. Reduced the XML file sizes
Best,
Saeed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #1865:
URL: https://github.com/apache/systemds/pull/1865#issuecomment-1640917047
It is just missing an exception on :
https://github.com/apache/systemds/blob/715b2fc523f700974ad1aa348d155553ed1bc52f/pom.xml#L636
for the path: src/test/scripts/functions/iogen/in/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] fathollahzadeh commented on pull request #1865: [SYSTEMDS][GIO] Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example
Posted by "fathollahzadeh (via GitHub)" <gi...@apache.org>.
fathollahzadeh commented on PR #1865:
URL: https://github.com/apache/systemds/pull/1865#issuecomment-1641019255
> It is just missing an exception on :
>
> https://github.com/apache/systemds/blob/715b2fc523f700974ad1aa348d155553ed1bc52f/pom.xml#L636
>
> for the path: src/test/scripts/functions/iogen/in/
can I add this line to pom.xml?
<exclude>src/test/scripts/functions/iogen/in/*/*/*</exclude>
or, should I find another way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org