[systemds] branch master updated: [SYSTEMDS-2712] Inferring CSV mdt bug fix

[ba...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

baunsgaard pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/systemds.git


The following commit(s) were added to refs/heads/master by this push:
     new f6cdfce  [SYSTEMDS-2712] Inferring CSV mdt bug fix
f6cdfce is described below

commit f6cdfce98d2d2f677426dbc525d396fb6d6c2eaa
Author: baunsgaard <ba...@tugraz.at>
AuthorDate: Mon Nov 2 15:21:20 2020 +0100

    [SYSTEMDS-2712] Inferring CSV mdt bug fix
    
    When reading CSV files and directly writing them to another file.
    The metadata it saves with indicate -1 cols and -1 rows.
    This is because it while reading the CSV does not know how big it is
    in default arguments.
    
    This commit fixes this, when the first read from disk is called, the
    metadata is changed on the matrix object after the read, to reflect the
    correct number of cols, rows and nnz.
---
 .../apache/sysds/runtime/controlprogram/caching/MatrixObject.java   | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/main/java/org/apache/sysds/runtime/controlprogram/caching/MatrixObject.java b/src/main/java/org/apache/sysds/runtime/controlprogram/caching/MatrixObject.java
index 4fd2b06..61bc5b8 100644
--- a/src/main/java/org/apache/sysds/runtime/controlprogram/caching/MatrixObject.java
+++ b/src/main/java/org/apache/sysds/runtime/controlprogram/caching/MatrixObject.java
@@ -462,6 +462,12 @@ public class MatrixObject extends CacheableData<MatrixBlock>
 			DataConverter.readMatrixFromHDFS(fname, iimd.getFileFormat(), rlen,
 			clen, mc.getBlocksize(), mc.getNonZeros(), getFileFormatProperties());
 		
+		if(iimd.getFileFormat() == FileFormat.CSV){
+			_metaData = _metaData instanceof MetaDataFormat ?
+				new MetaDataFormat(newData.getDataCharacteristics(), iimd.getFileFormat()) :
+				new MetaData(newData.getDataCharacteristics());
+		}
+		
 		setHDFSFileExists(true);
 		
 		//sanity check correct output