You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2021/04/24 11:58:42 UTC
[GitHub] [systemds] lederb opened a new pull request #1239: Python api/frame support
lederb opened a new pull request #1239:
URL: https://github.com/apache/systemds/pull/1239
Hi,
I added some basic Frame functionality for the PythonAPI.
Focused on:
- Frame generation from pandas.DataFrame
- write and read from csv incl. tests
- rbind, cbind incl. tests
- transform_encode and transform_apply incl. tests
- hyperband test
Currently all those tests and the previous ones (~110 in total) are passing on my side
Sadly I forgot to turn off my auto-formatter, so right now it looks like there were quite some changes especially in
`src/main/python/systemds/context/systemds_context.py`
real changes are pretty much only in read()
`src/main/python/systemds/operator/operation_node.py`
real changes in:
- rbind(), cbind()
- some parser conditions added for OutputType.FRAME
- some checks/assertions
if the formatting changes are an issue for those two files, let me know, then I'll try to find a proper way of fixing those.
just wanted to add an initial pull-request so it can already be inspected.
cheers,
Bernhard Leder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] lederb commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
lederb commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-826326729
ok thanks. I already added it to the setup.py for the wheel but not to the workflow. thx for the response, even though it's sunday :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-828368500
Hi @lederb
Thanks for your contribution :1st_place_medal: , while merging i moved the dateset for the test into our resources folder,
and regenerated our builtin files. This conclude the coding project in AMLS with success.
I like that you added a few tests for transform encode and decode as well, going over the scope of the initially assigned project.
the commit currently is associated with an email that is not claimed to your account, I would suggest you do that on your GitHub account to have full credit later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] lederb commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
lederb commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-826309953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard closed pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
Baunsgaard closed pull request #1239:
URL: https://github.com/apache/systemds/pull/1239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-826325153
> I saw that the build failed because of some missing license headers, sorry for that but I think I found the missing ones.
>
> Reverting the formatting changes using black didn't work, only would have changed some lines additionally.
> Thus I reverted the 3 files in question back to the master branch, then checked out my other work in progress branch for those files and went through the diffs and changes one by one.
>
> I would also have a question about, how your test-suite / CI-yaml configs are created? Because I only saw that the tests are failing because there is no pandas package installed, when looking at the python.yaml I only see:
>
> ` - name: Install pip Dependencies`
> `run: pip install numpy py4j wheel scipy sklearn requests`
You can add pandas to that line, it will install it for the tests.
also add it as a dependency for the python package in general:
https://github.com/apache/systemds/blob/master/src/main/python/setup.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] lederb commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
lederb commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-828456254
Thanks for approving the PR and also for noting the email mix-up.
Do you think it would be a bad idea if I amend my last commit, change the authorship and force push it to my repo?
Could this mess anything up?
Otherwise, I just added my work email to the account.
cheers,
Bernhard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-826325153
> I saw that the build failed because of some missing license headers, sorry for that but I think I found the missing ones.
>
> Reverting the formatting changes using black didn't work, only would have changed some lines additionally.
> Thus I reverted the 3 files in question back to the master branch, then checked out my other work in progress branch for those files and went through the diffs and changes one by one.
>
> I would also have a question about, how your test-suite / CI-yaml configs are created? Because I only saw that the tests are failing because there is no pandas package installed, when looking at the python.yaml I only see:
>
> ` - name: Install pip Dependencies`
> `run: pip install numpy py4j wheel scipy sklearn requests`
You can add pandas to that line, it will install it for the tests.
also add it as a dependency for the python package in general:
https://github.com/apache/systemds/blob/master/src/main/python/setup.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] lederb commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
lederb commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-826309953
I saw that the build failed because of some missing license headers, sorry for that but I think I found the missing ones.
Reverting the formatting changes using black didn't work, only would have changed some lines additionally.
Thus I reverted the 3 files in question back to the master branch, then checked out my other work in progress branch for those files and went through the diffs and changes one by one.
I would also have a question about, how your test-suite / CI-yaml configs are created? Because I only saw that the tests are failing because there is no pandas package installed, when looking at the python.yaml I only see:
` - name: Install pip Dependencies`
`run: pip install numpy py4j wheel scipy sklearn requests`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1239: [PythonAPI] frame support
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1239:
URL: https://github.com/apache/systemds/pull/1239#issuecomment-828759763
> Thanks for approving the PR and also for noting the email mix-up.
> Do you think it would be a bad idea if I amend my last commit, change the authorship and force push it to my repo?
> Could this mess anything up?
> Otherwise, I just added my work email to the account.
>
> cheers,
> Bernhard
It seems like you solved it , since the commit have been claimed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org