[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1397: [SYSTEMDS-3143] Frame rmempty instruction

[GitBox <gi...@apache.org>]

Shafaq-Siddiqi commented on a change in pull request #1397:
URL: https://github.com/apache/systemds/pull/1397#discussion_r721461423



##########
File path: src/main/java/org/apache/sysds/runtime/matrix/data/FrameBlock.java
##########
@@ -2345,6 +2384,77 @@ public FrameBlock replaceOperations(String pattern, String replacement){
 		return ret;
 	}
 
+	public  FrameBlock removeEmptyOperations(boolean rows, boolean emptyReturn, MatrixBlock select) {
+		if( rows )
+			return removeEmptyRows(select, emptyReturn);
+		else //cols
+			return removeEmptyColumns(select, emptyReturn);
+	}
+
+	private FrameBlock removeEmptyRows(MatrixBlock select, boolean emptyReturn) {
+		FrameBlock ret = new FrameBlock(_schema, _colnames);
+
+		for(int i = 0; i < _numRows; i++) {
+			boolean isEmpty = true;
+			Object[] row = new Object[getNumColumns()];
+
+			for(int j = 0; j < getNumColumns(); j++) {
+				Array colData = _coldata[j].clone();
+				row[j] = colData.get(i);
+				ValueType type = _schema[j];
+				isEmpty = isEmpty && (ArrayUtils.contains(new double[]{0.0, Double.NaN}, UtilFunctions.objectToDoubleSafe(type, colData.get(i))));

Review comment:
       UtilFunctions.objectToDoubleSafe() converts a string representation "null" to zero and then removes it, but it will not work with the original null representation that we get while reading a file with missing values.  Instead of creating a new representation for "null" write a frame in your java test suite and read it in dml by setting the "naString" flag or read an external dataset.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org