[jira] [Created] (ARROW-10174) [Java] Reading of Dictionary encoded struct vector fails

["Benjamin Wilhelm (Jira)" <ji...@apache.org>]

Benjamin Wilhelm created ARROW-10174:
----------------------------------------

             Summary: [Java] Reading of Dictionary encoded struct vector fails 
                 Key: ARROW-10174
                 URL: https://issues.apache.org/jira/browse/ARROW-10174
             Project: Apache Arrow
          Issue Type: Bug
          Components: Java
    Affects Versions: 1.0.1
            Reporter: Benjamin Wilhelm


Write an index vector and a dictionary with a dictionary vector of the type {{Struct}} using an {{ArrowStreamWriter}}. Reading this again fails with an exception.

Code to reproduce:

{code:java}
final RootAllocator allocator = new RootAllocator();

// Create the dictionary
final StructVector dict = StructVector.empty("Dict", allocator);
final NullableStructWriter dictWriter = dict.getWriter();
final IntWriter dictA = dictWriter.integer("a");
final IntWriter dictB = dictWriter.integer("b");
for (int i = 0; i < 3; i++) {
	dictWriter.start();
	dictA.writeInt(i);
	dictB.writeInt(i);
	dictWriter.end();
}
dict.setValueCount(3);
final Dictionary dictionary = new Dictionary(dict, new DictionaryEncoding(1, false, null));

// Create the vector
final Random random = new Random();
final StructVector vector = StructVector.empty("Dict", allocator);
final NullableStructWriter vectorWriter = vector.getWriter();
final IntWriter vectorA = vectorWriter.integer("a");
final IntWriter vectorB = vectorWriter.integer("b");
for (int i = 0; i < 10; i++) {
	int v = random.nextInt(3);
	vectorWriter.start();
	vectorA.writeInt(v);
	vectorB.writeInt(v);
	vectorWriter.end();
}
vector.setValueCount(10);

// Encode the vector using the dictionary
final IntVector indexVector = (IntVector) DictionaryEncoder.encode(vector, dictionary);

// Write the vector to out
final ByteArrayOutputStream out = new ByteArrayOutputStream();
final VectorSchemaRoot root = new VectorSchemaRoot(Collections.singletonList(indexVector.getField()),
		Collections.singletonList(indexVector));
final ArrowStreamWriter writer = new ArrowStreamWriter(root, new MapDictionaryProvider(dictionary),
		Channels.newChannel(out));
writer.start();
writer.writeBatch();
writer.end();

// Read the vector from out
try (final ArrowStreamReader reader = new ArrowStreamReader(new ByteArrayInputStream(out.toByteArray()),
		allocator)) {
	reader.loadNextBatch();
	final VectorSchemaRoot readRoot = reader.getVectorSchemaRoot();
	final FieldVector readIndexVector = readRoot.getVector(0);

	// Get the dictionary and decode
	final Map<Long, Dictionary> readDictionaryMap = reader.getDictionaryVectors();
	final Dictionary readDictionary = readDictionaryMap.get(readIndexVector.getField().getDictionary().getId());
	final ValueVector readVector = DictionaryEncoder.decode(readIndexVector, readDictionary);
}
{code}

Exception:
{code}
java.lang.IllegalArgumentException: not all nodes and buffers were consumed. nodes: [ArrowFieldNode [length=3, nullCount=0], ArrowFieldNode [length=3, nullCount=0]] buffers: [ArrowBuf[21], address:140118352739688, length:1, ArrowBuf[22], address:140118352739696, length:12, ArrowBuf[23], address:140118352739712, length:1, ArrowBuf[24], address:140118352739720, length:12]
	at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:63)
	at org.apache.arrow.vector.ipc.ArrowReader.load(ArrowReader.java:241)
	at org.apache.arrow.vector.ipc.ArrowReader.loadDictionary(ArrowReader.java:232)
	at org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:129)
	at com.knime.AppTest.testDictionaryStruct(AppTest.java:83)
{code}

If I see it corretly the error happens in {{DictionaryUtilities#toMessageFormat}}. If a dictionary encoded vector is encountered still the children of the memory format field are used (none because this is Int). However, the children of the field of the dictionary vector should be mapped to the message format and set as children.

I can create a fix and open a pull request.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)