You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "shieru1214 (via GitHub)" <gi...@apache.org> on 2026/09/16 18:00:13 UTC

[PR] [SYSTEMDS-3971] Improve Scuro test coverage [systemds]

shieru1214 opened a new pull request, #2615:
URL: https://github.com/apache/systemds/pull/2615

   
   This change adds test coverage for methods related to **persistence** and **results retrieval** in `UnimodalOptimizer`.
   
   ### Persistence
   
   - `store_results`:It saves the optimizer results to a file and stores execution statistics separately, including worker statistics, node reuse, and runtime.
   - `load_results`: It loads previously saved results and can be reused without running the search again.
   - `_count_results_by_modality`: It counts the existing results for each modality. This count is used as checkpoint progress when resuming the optimization.
   - `resume_from_checkpoint`: It can restore the previous result from a checkpoint. If no checkpoint is available, the current results remain unchanged.
   
   ### Results Retrieval
   
   - `get_k_best_results`: It selects the top `k` results according to their scores and returns both the results and their corresponding data. The tests cover using existing cached data and rebuilding the data through the result DAG when the cache is empty.
   - `get_dag_by_id`: It searches a list of DAGs by `dag_id`. It returns the matching DAG or `None` when the ID is not found.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3971] Improve Scuro test coverage [systemds]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2615:
URL: https://github.com/apache/systemds/pull/2615#issuecomment-5940746268

   ## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2615?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 75.66%. Comparing base ([`8c7d556`](https://app.codecov.io/gh/apache/systemds/commit/8c7d556da8fc218704c26313d81c3f058910228e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b3134b4`](https://app.codecov.io/gh/apache/systemds/commit/b3134b4f985abb8706b6b64dc2e52fff4804fba9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 19 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #2615       +/-   ##
   =============================================
   + Coverage     71.81%   75.66%    +3.84%     
   =============================================
     Files          2075      436     -1639     
     Lines        222176    25465   -196711     
     Branches      38264        0    -38264     
   =============================================
   - Hits         159559    19268   -140291     
   + Misses        51561     6197    -45364     
   + Partials      11056        0    -11056     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/systemds/pull/2615/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [java](https://app.codecov.io/gh/apache/systemds/pull/2615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [python](https://app.codecov.io/gh/apache/systemds/pull/2615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.66% <100.00%> (+0.70%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/systemds/pull/2615?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3971] Improve Scuro test coverage [systemds]

Posted by "shieru1214 (via GitHub)" <gi...@apache.org>.
shieru1214 commented on PR #2615:
URL: https://github.com/apache/systemds/pull/2615#issuecomment-6486896736

   This change adds 12 tests for representation_dag.py, covering DAG validation, construction, execution, and helper functions. 
   
   ### DAG Validation
   
   I added tests for `validate()`. A connected DAG should pass, while an empty DAG, a missing root, a reference to an input that does not exist, a cycle, and a self-loop should all fail. I also check that a node not connected to the root is removed when the DAG is created. These tests make the expected behaviour for incomplete DAGs more explicit.
   
   ### DAG Construction
   
   This part covers both `RepresentationDAGBuilder` and `CSEAwareDAGBuilder`. I check that neither builder accepts a root that does not exist. For the CSE builder, I test when nodes should be reused: the same input, operator, and `window_size` give the same node, but changing `window_size` from 2 to 4 gives a different one. I also check that building the same root again returns the same DAG object, while giving it a new `dag_id` creates a new DAG object.
   
   ### DAG Execution
   
   I run a small DAG where one modality goes through `WindowAggregation` and is then combined with another modality using `Concatenation`. The test checks that the root returns a `TransformedModality` with shape `(6, 6)`. Two other tests check that a fusion operator needing training cannot run directly through `execute()`, and that the node cache stores only the intermediate node. On the second run, that node is not stored again. These tests check the execution path and cache behaviour, but do not compare the output values.
   
   ### Helper Functions
   
   The modality lookup test checks that it returns the requested object when several objects share a `modality_id`, and returns `None` when the ID is missing. The pushdown parameter test checks that an aggregation object is created only when `_pushdown_aggregation` is present. Finally, I check that `pushdown_aggregation` leaves the nodes and their connections unchanged when `WindowAggregation` does not support pushdown. These tests do not check the numerical result of aggregation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3971] Improve Scuro test coverage [systemds]

Posted by "christinadionysio (via GitHub) via dev" <de...@systemds.apache.org>.
christinadionysio merged PR #2615:
URL: https://github.com/apache/systemds/pull/2615


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3971] Improve Scuro test coverage [systemds]

Posted by "shieru1214 (via GitHub)" <gi...@apache.org>.
shieru1214 commented on PR #2615:
URL: https://github.com/apache/systemds/pull/2615#issuecomment-5881391015

   
   This change improves the test coverage of `hyperparameter_tuner.py`. The added tests focus on three parts: search space construction, trial parameter, and tuning result handling.
   
   ### 1. Search space construction
   
   This part converts the parameter ranges declared by operators into search spaces used by Optuna. `_build_param_specs` is the entry function and was already fully covered, so I mainly added tests for its three helpers. The tests cover the input types handled by `_param_values_to_spec`, the input stats created from `window_size`, and the expansion and narrowing of nested aggregation parameters.
   
   ### 2. Trial parameter 
   
   The first part flattens the parameters, while this part puts the values selected by Optuna back into the correct DAG nodes. It mainly covers `_apply_trial_params_to_node`, `_apply_pushdown_trial_params`, `_materialize_node_params`, and the two operation checks. I added tests for pushed-down aggregations and `AggregatedRepresentation`, and also checked parameter placement, unchanged base parameters, non-class inputs, import failures, leaf nodes, and empty parameters.
   
   ### 3. Tuning result handling
   
   This part covers how tuning results are stored and read back. It includes `add_result`, `setup_mm`, `get_k_best_dags`, and `get_k_best_results`. The new tests check multimodal result storage, skipped `None` results, and the two `optimize_unimodal` cases. They also check that `best_params` are applied to a rebuilt DAG, the leaf and root are kept correctly, the original DAG is unchanged, and empty or multiple results are handled.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3971] Improve Scuro test coverage [systemds]

Posted by "shieru1214 (via GitHub)" <gi...@apache.org>.
shieru1214 commented on PR #2615:
URL: https://github.com/apache/systemds/pull/2615#issuecomment-5943790722

   This change adds  tests for `multimodal_optimizer.py`.  The new tests cover three parts: fusion DAG generation, constructor setup, and the optimize loop.
   
   ### Fusion DAG Generation
   
   Before this change, the three methods that generate the fusion search space had no active tests.
   
   - `_generate_modality_combinations`: `test_modality_combinations_respect_min_max_modalities` uses three modalities and checks the exact subsets for `max_modalities` values of 2, 3, and 5. This also checks the case where the maximum is larger than the number of available modalities.
   - `_generate_representation_combinations`: `test_representation_combinations_pick_one_per_modality` uses one modality with two representations and another with three. It checks all six possible combinations.
   - `_generate_fusion_dags`: `test_fusion_dags_contain_every_selected_representation` checks every generated DAG. It checks that the selected modality and representation leaves are present, the number of fusion nodes is correct, each fusion node has two inputs, and the root is a fusion node.
   - `test_fusion_dags_use_every_fusion_operator` checks that both configured operators, `Concatenation` and `Average`, are used. This is tested separately because a DAG can have the correct structure but still miss one operator.
   
   
   ### Constructor Setup
   
   The generator tests already called `__init__`, but they did not directly check the values created during construction. I added three tests for this part.
   
   - `test_min_modalities_clamped_to_two` checks that `min_modalities=1` is changed to `2`. This prevents single-modality DAGs from being treated as fusion DAGs.
   - `test_max_modalities_defaults_to_number_of_modalities` checks that the default maximum is the number of available modalities.
   - `test_k_best_representations_hold_the_cached_data` covers `_extract_k_best_representations`. It checks that every modality is stored by its ID and that the optimizer keeps the cached representation data returned by the unimodal results, instead of the score records.
   
   ### Optimize Loop
   
   Before this change, none of the tests called `optimize`. I added three tests for the main rules of this loop.
   
   - `test_stops_at_max_combinations` uses three modalities and sets `max_combinations=5`. It checks that `_evaluate_dag` is called exactly five times and that five results are stored.
   - `test_drops_failed_evaluations` makes every second evaluation return `None`. It checks that six evaluations are attempted but only three results are stored. Failed evaluations still count towards the limit, and the loop continues after them.
   - `test_keeps_results_and_budget_per_task` uses two tasks with a limit of two. It checks that each task receives two evaluations and that every result is stored under the correct task name.
   
   、
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org