[GitHub] [systemds] kev-inn commented on pull request #1665: [SYSTEMDS-3405] Write matrices and frames at site for federated write

[GitBox <gi...@apache.org>]

kev-inn commented on PR #1665:
URL: https://github.com/apache/systemds/pull/1665#issuecomment-1200477730

   There is a minor problem I don't know how to best fix. I need to select a path at the sites for the workers to write their partition, I am currently choosing to create a (most likely) unique filename and write into the LOCAL_TEMP_DIR (defined by the configuration). This usually works, but the federated python testcases have it set to `/tmp/systemds` which is apparently not available for our git runners. @Baunsgaard maybe we could change the configuration for the tests?
   
   The problems mostly impact our testcases, but I also don't really have a favorite from a user side.
   Other choices for where to write and downsides:
   - current working directory (cwd): the cwd of our testcases is the root, therefore we would clutter it full of partitions
   - the scratch space: will be deleted when the worker is killed and in our case, because we just start the new workers from the same JVM, also when the testcase finishes. Therefore, we can't check the results.
   - A new configuration directory: pretty clean solution IMO, but would only be used by workers and I don't want to add it, except if we agree it is necessary
   
   The other aspects are finished and this PR is ready for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org