You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2021/09/21 22:24:20 UTC
[GitHub] [systemds] OlgaOvcharenko opened a new pull request #1397: [WIP][SYSTEMDS-3143] Frame rmempty instruction
OlgaOvcharenko opened a new pull request #1397:
URL: https://github.com/apache/systemds/pull/1397
This PR adds frame rmempty instruction, empty values are 0.0, FALSE, null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard closed pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Baunsgaard closed pull request #1397:
URL: https://github.com/apache/systemds/pull/1397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] OlgaOvcharenko commented on pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
OlgaOvcharenko commented on pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#issuecomment-962014642
@Shafaq-Siddiqi Could you please review it once you have time. The tests were modified (NAs handling), moreover, I believe that select vector and margin are supported, so there is no need for TODO. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#issuecomment-962045558
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#discussion_r721461423
##########
File path: src/main/java/org/apache/sysds/runtime/matrix/data/FrameBlock.java
##########
@@ -2345,6 +2384,77 @@ public FrameBlock replaceOperations(String pattern, String replacement){
return ret;
}
+ public FrameBlock removeEmptyOperations(boolean rows, boolean emptyReturn, MatrixBlock select) {
+ if( rows )
+ return removeEmptyRows(select, emptyReturn);
+ else //cols
+ return removeEmptyColumns(select, emptyReturn);
+ }
+
+ private FrameBlock removeEmptyRows(MatrixBlock select, boolean emptyReturn) {
+ FrameBlock ret = new FrameBlock(_schema, _colnames);
+
+ for(int i = 0; i < _numRows; i++) {
+ boolean isEmpty = true;
+ Object[] row = new Object[getNumColumns()];
+
+ for(int j = 0; j < getNumColumns(); j++) {
+ Array colData = _coldata[j].clone();
+ row[j] = colData.get(i);
+ ValueType type = _schema[j];
+ isEmpty = isEmpty && (ArrayUtils.contains(new double[]{0.0, Double.NaN}, UtilFunctions.objectToDoubleSafe(type, colData.get(i))));
Review comment:
UtilFunctions.objectToDoubleSafe() converts a string representation "null" to zero and then removes it, but it will not work with the original null representation that we get while reading a file with missing values. Instead of creating a new representation for "null" write a frame in your java test suite and read it in dml by setting the "naString" flag or read an external dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] OlgaOvcharenko commented on pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
OlgaOvcharenko commented on pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#issuecomment-962014642
@Shafaq-Siddiqi Could you please review it once you have time. The tests were modified (NAs handling), moreover, I believe that select vector and margin are supported, so there is no need for TODO. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard closed pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Baunsgaard closed pull request #1397:
URL: https://github.com/apache/systemds/pull/1397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#issuecomment-962045558
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] OlgaOvcharenko commented on pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
OlgaOvcharenko commented on pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#issuecomment-962014642
@Shafaq-Siddiqi Could you please review it once you have time. The tests were modified (NAs handling), moreover, I believe that select vector and margin are supported, so there is no need for TODO. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard closed pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Baunsgaard closed pull request #1397:
URL: https://github.com/apache/systemds/pull/1397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#issuecomment-962045558
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#discussion_r721462372
##########
File path: src/main/java/org/apache/sysds/runtime/matrix/data/FrameBlock.java
##########
@@ -2345,6 +2384,77 @@ public FrameBlock replaceOperations(String pattern, String replacement){
return ret;
}
+ public FrameBlock removeEmptyOperations(boolean rows, boolean emptyReturn, MatrixBlock select) {
+ if( rows )
+ return removeEmptyRows(select, emptyReturn);
+ else //cols
+ return removeEmptyColumns(select, emptyReturn);
+ }
+
+ private FrameBlock removeEmptyRows(MatrixBlock select, boolean emptyReturn) {
+ FrameBlock ret = new FrameBlock(_schema, _colnames);
+
+ for(int i = 0; i < _numRows; i++) {
+ boolean isEmpty = true;
+ Object[] row = new Object[getNumColumns()];
+
+ for(int j = 0; j < getNumColumns(); j++) {
+ Array colData = _coldata[j].clone();
+ row[j] = colData.get(i);
+ ValueType type = _schema[j];
+ isEmpty = isEmpty && (ArrayUtils.contains(new double[]{0.0, Double.NaN}, UtilFunctions.objectToDoubleSafe(type, colData.get(i))));
+ }
+
+ if((!isEmpty && select == null) || (select != null && select.getValue(i, 0) == 1)) {
Review comment:
Please add a TODO for the remaining functionalities like support for select vector and margin = "cols"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org