You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "martin-traverse (via GitHub)" <gi...@apache.org> on 2025/02/26 02:52:30 UTC
[PR] Feature / Produce Avro core data types out of Arrow VSR [arrow-java]
martin-traverse opened a new pull request, #638:
URL: https://github.com/apache/arrow-java/pull/638
## What's Changed
Per discussion in #615 , here is a first take on the core producers to generate Avro data from Arrow vectors. There are a few points I'd like to clarify before going further:
* Nullability. Avro only understands nullable types as unions but that is not normally how they will be if the data comes from other sources. I have added a special NullableProducer to handle nullable vectors which are not unions. We will need something equivalent in the consumers and probably a setting in the AvroToArrowConfig to control it on read, defaulting to current behaviour. I have also added special handling for nullable unions, because unions can't be nested (i.e. you can't nest "type | null" as a type inside a union). I can add consumers to handle both (unions and regular types) for review, if that sounds right? At the moment the schema for nullable fields gets quite mangled on a round trip!
* Arrow has a lot more types than Avro, at the level of minor / vector types. Going Avro -> Arrow we just pick the direct equivalent. Going Arrow -> Avro, we could cast silently if there is no loss of precision. E.g. TinyInt and SmallInt -> Int and so on. For types like e.g. Decimal256 and LargeVarChar we could write out safely but would need support in the consumers to read back the wider types. I could start by adding the safe conversions now and we could come back to the wide types in a later PR maybe?
* Type information is inferred from the list of vectors, using minor types. We'll also need to generate the Avro schema, the input for that would be a list of fields. I haven't done it yet but will do if that sounds right.
* Dictionary encoding for enums not implemented yet, I'll add it if the rest looks good. Caveat is that dictionaries must be fixed before the encoding starts if we are writing out the whole file in one go (i.e. if the Avro schema is at the start of the container file). If the schema is saved separately that limitation need not apply, we could provide the schema once encoding is finished.
Please do let me know if this is going in the right direction + any comments. If it is I will add the missing pieces and start the exhaustive test coverage to mirror the consumers. Once it's done this PR should get us to the point where we can round trip the contents of an individual block for most data types, but it does not address the container format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2015644238
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroListProducer.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces array values from a {@link ListVector}, writes data to an avro encoder.
+ */
+public class AvroListProducer extends BaseAvroProducer<ListVector> {
+
+ private final Producer<? extends FieldVector> delegate;
+
+ /** Instantiate an AvroListProducer. */
+ public AvroListProducer(ListVector vector, Producer<? extends FieldVector> delegate) {
+ super(vector);
+ this.delegate = delegate;
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+
+ int startOffset = vector.getOffsetBuffer().getInt(currentIndex * (long) Integer.BYTES);
+ int endOffset = vector.getOffsetBuffer().getInt((currentIndex + 1) * (long) Integer.BYTES);
+ int nItems = endOffset - startOffset;
+
+ encoder.writeArrayStart();
+ encoder.setItemCount(nItems);
+
+ for (int i = 0; i < nItems; i++) {
+ encoder.startItem();
+ delegate.produce(encoder);
+ }
+
+ encoder.writeArrayEnd();
+ currentIndex++;
+ }
+
+ // Do not override skipNull(), delegate will not have an entry if the list is null
+
+ @Override
+ public void setPosition(int index) {
+ if (index < 0 || index > vector.getValueCount()) {
Review Comment:
Ah, I see your comment below now. It might be worth noting this somewhere (perhaps the docstring of the base class?)
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroListProducer.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces array values from a {@link ListVector}, writes data to an avro encoder.
+ */
+public class AvroListProducer extends BaseAvroProducer<ListVector> {
+
+ private final Producer<? extends FieldVector> delegate;
+
+ /** Instantiate an AvroListProducer. */
+ public AvroListProducer(ListVector vector, Producer<? extends FieldVector> delegate) {
+ super(vector);
+ this.delegate = delegate;
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+
+ int startOffset = vector.getOffsetBuffer().getInt(currentIndex * (long) Integer.BYTES);
+ int endOffset = vector.getOffsetBuffer().getInt((currentIndex + 1) * (long) Integer.BYTES);
+ int nItems = endOffset - startOffset;
+
+ encoder.writeArrayStart();
+ encoder.setItemCount(nItems);
+
+ for (int i = 0; i < nItems; i++) {
+ encoder.startItem();
+ delegate.produce(encoder);
+ }
+
+ encoder.writeArrayEnd();
+ currentIndex++;
+ }
+
+ // Do not override skipNull(), delegate will not have an entry if the list is null
+
+ @Override
+ public void setPosition(int index) {
+ if (index < 0 || index > vector.getValueCount()) {
Review Comment:
Should these checks be `index >= getValueCount()`?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDenseUnionProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.DenseUnionVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <ul>
+ * <li>ArrowType.Null --> NULL
+ * <li>ArrowType.Bool --> BOOLEAN
+ * <li>ArrowType.Int(64 bit, unsigned 32 bit) --> LONG
+ * <li>ArrowType.Int(signed 32 bit, < 32 bit) --> INT
+ * <li>ArrowType.FloatingPoint(double) --> DOUBLE
+ * <li>ArrowType.FloatingPoint(single, half) --> FLOAT
+ * <li>ArrowType.Utf8 --> STRING
+ * <li>ArrowType.LargeUtf8 --> STRING
+ * <li>ArrowType.Binary --> BYTES
+ * <li>ArrowType.LargeBinary --> BYTES
+ * <li>ArrowType.FixedSizeBinary --> FIXED
+ * <li>ArrowType.Decimal --> decimal (FIXED)
+ * <li>ArrowType.Date --> date (INT)
+ * <li>ArrowType.Time (MILLI) --> time-millis (INT)
+ * <li>ArrowType.Time (SEC | MICRO | NANO) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ != NULL) --> time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ != NULL) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ != NULL) --> time-millis (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ == NULL) --> local-time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ == NULL) --> local-time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ == NULL) --> local-time-millis (LONG)
+ * <li>ArrowType.Duration --> duration (FIXED)
+ * <li>ArrowType.Interval --> duration (FIXED)
+ * <li>ArrowType.Struct --> record
+ * <li>ArrowType.List --> array
+ * <li>ArrowType.LargeList --> array
+ * <li>ArrowType.FixedSizeList --> array
+ * <li>ArrowType.Map --> map
+ * <li>ArrowType.Union --> union
+ * </ul>
+ *
+ * <p>Nullable fields are represented as a union of [null | base-type]. Special treatment is given
+ * to nullability of unions - a union is considered nullable if the union field is nullable or any
Review Comment:
Can we update the docstring too?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <table>
+ * <thead><tr><th>Arrow type</th><th>Avro encoding</th></tr></thead>
+ * <tbody>
+ * <tr><td>ArrowType.Null</td><td>NULL</td></tr>
+ * <tr><td>ArrowType.Bool</td><td>BOOLEAN</td></tr>
+ * <tr><td>ArrowType.Int(64 bit, unsigned 32 bit)</td><td>LONG</td></tr>
+ * <tr><td>ArrowType.Int(signed 32 bit, < 32 bit)</td><td>INT</td></tr>
+ * <tr><td>ArrowType.FloatingPoint(double)</td><td>DOUBLE</td></tr>
+ * <tr><td>ArrowType.FloatingPoint(single, half)</td><td>FLOAT</td></tr>
+ * <tr><td>ArrowType.Utf8</td><td>STRING</td></tr>
+ * <tr><td>ArrowType.LargeUtf8</td><td>STRING</td></tr>
+ * <tr><td>ArrowType.Binary</td><td>BYTES</td></tr>
+ * <tr><td>ArrowType.LargeBinary</td><td>BYTES</td></tr>
+ * <tr><td>ArrowType.FixedSizeBinary</td><td>FIXED</td></tr>
+ * <tr><td>ArrowType.Decimal</td><td>decimal (FIXED)</td></tr>
+ * <tr><td>ArrowType.Date</td><td>date (INT)</td></tr>
+ * <tr><td>ArrowType.Time (SEC | MILLI)</td><td>time-millis (INT)</td></tr>
+ * <tr><td>ArrowType.Time (MICRO | NANO)</td><td>time-micros (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (NANOSECONDS, TZ != NULL)</td><td>time-nanos (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MICROSECONDS, TZ != NULL)</td><td>time-micros (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ != NULL)</td><td>time-millis (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (NANOSECONDS, TZ == NULL)</td><td>local-time-nanos (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MICROSECONDS, TZ == NULL)</td><td>local-time-micros (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ == NULL)</td><td>local-time-millis (LONG)</td></tr>
+ * <tr><td>ArrowType.Duration</td><td>duration (FIXED)</td></tr>
+ * <tr><td>ArrowType.Interval</td><td>duration (FIXED)</td></tr>
+ * <tr><td>ArrowType.Struct</td><td>record</td></tr>
+ * <tr><td>ArrowType.List</td><td>array</td></tr>
+ * <tr><td>ArrowType.LargeList</td><td>array</td></tr>
+ * <tr><td>ArrowType.FixedSizeList</td><td>array</td></tr>
+ * <tr><td>ArrowType.Map</td><td>map</td></tr>
+ * <tr><td>ArrowType.Union</td><td>union</td></tr>
+ * </tbody>
+ * </table>
+ *
+ * <p>Nullable fields are represented as a union of [null | base-type]. Special treatment is given
+ * to nullability of unions - a union is considered nullable if the union field is nullable or any
+ * of its child fields are nullable. The schema for a nullable union will always contain a null
+ * type,none of the direct child types will be nullable.
+ *
+ * <p>List fields must contain precisely one child field, which may be nullable. Map fields are
+ * represented as a list of structs, where the struct fields are "key" and "value". The key field
+ * must always be of type STRING (Utf8) and cannot be nullable. The value can be of any type and
+ * may be nullable. Record types must contain at least one child field and cannot contain multiple
+ * fields with the same name
+ *
+ * @param arrowFields The arrow fields used to generate the Avro schema
+ * @param typeName Name of the top level Avro record type
+ * @param namespace Namespace of the top level Avro record type
+ * @return An Avro record schema for the given list of fields, with the specified name and
+ * namespace
+ */
+ public static Schema createAvroSchema(
+ List<Field> arrowFields, String typeName, String namespace) {
+ SchemaBuilder.RecordBuilder<Schema> assembler =
+ SchemaBuilder.record(typeName).namespace(namespace);
+ return buildRecordSchema(assembler, arrowFields, namespace);
+ }
+
+ /** Overload provided for convenience, sets namespace = null. */
+ public static Schema createAvroSchema(List<Field> arrowFields, String typeName) {
+ return createAvroSchema(arrowFields, typeName, null);
+ }
+
+ /** Overload provided for convenience, sets name = GENERIC_RECORD_TYPE_NAME. */
+ public static Schema createAvroSchema(List<Field> arrowFields) {
+ return createAvroSchema(arrowFields, GENERIC_RECORD_TYPE_NAME);
+ }
+
+ private static <T> T buildRecordSchema(
+ SchemaBuilder.RecordBuilder<T> builder, List<Field> fields, String namespace) {
+ if (fields.isEmpty()) {
+ throw new IllegalArgumentException("Record field must have at least one child field");
+ }
+ SchemaBuilder.FieldAssembler<T> assembler = builder.namespace(namespace).fields();
+ for (Field field : fields) {
+ assembler = buildFieldSchema(assembler, field, namespace);
+ }
+ return assembler.endRecord();
+ }
+
+ private static <T> SchemaBuilder.FieldAssembler<T> buildFieldSchema(
+ SchemaBuilder.FieldAssembler<T> assembler, Field field, String namespace) {
+
+ SchemaBuilder.FieldTypeBuilder<T> builder = assembler.name(field.getName()).type();
+
+ // Nullable unions need special handling, since union types cannot be directly nested
+ if (field.getType().getTypeID() == ArrowType.ArrowTypeID.Union) {
+ boolean unionNullable = field.getChildren().stream().anyMatch(Field::isNullable);
+ if (unionNullable) {
+ SchemaBuilder.UnionAccumulator<SchemaBuilder.NullDefault<T>> union =
+ builder.unionOf().nullType();
+ return addTypesToUnion(union, field.getChildren(), namespace).nullDefault();
+ } else {
+ Field headType = field.getChildren().get(0);
+ List<Field> tailTypes = field.getChildren().subList(1, field.getChildren().size());
+ SchemaBuilder.UnionAccumulator<SchemaBuilder.FieldDefault<T, ?>> union =
+ buildUnionFieldSchema(builder.unionOf(), headType, namespace);
+ return addTypesToUnion(union, tailTypes, namespace).noDefault();
+ }
+ } else if (field.isNullable()) {
+ return buildBaseFieldSchema(builder.nullable(), field, namespace);
+ } else {
+ return buildBaseFieldSchema(builder, field, namespace);
+ }
+ }
+
+ private static <T> T buildArraySchema(
+ SchemaBuilder.ArrayBuilder<T> builder, Field listField, String namespace) {
+ if (listField.getChildren().size() != 1) {
+ throw new IllegalArgumentException("List field must have exactly one child field");
+ }
+ Field itemField = listField.getChildren().get(0);
+ return buildTypeSchema(builder.items(), itemField, namespace);
+ }
+
+ private static <T> T buildMapSchema(
+ SchemaBuilder.MapBuilder<T> builder, Field mapField, String namespace) {
+ if (mapField.getChildren().size() != 1) {
+ throw new IllegalArgumentException("Map field must have exactly one child field");
+ }
+ Field entriesField = mapField.getChildren().get(0);
+ if (mapField.getChildren().size() != 1) {
+ throw new IllegalArgumentException("Map entries must have exactly two child fields");
+ }
+ Field keyField = entriesField.getChildren().get(0);
+ Field valueField = entriesField.getChildren().get(1);
+ if (keyField.getType().getTypeID() != ArrowType.ArrowTypeID.Utf8 || keyField.isNullable()) {
+ throw new IllegalArgumentException(
+ "Map keys must be of type string and cannot be nullable for conversion to Avro");
+ }
+ return buildTypeSchema(builder.values(), valueField, namespace);
+ }
+
+ private static <T> T buildTypeSchema(
+ SchemaBuilder.TypeBuilder<T> builder, Field field, String namespace) {
+
+ // Nullable unions need special handling, since union types cannot be directly nested
+ if (field.getType().getTypeID() == ArrowType.ArrowTypeID.Union) {
+ boolean unionNullable = field.getChildren().stream().anyMatch(Field::isNullable);
+ if (unionNullable) {
+ SchemaBuilder.UnionAccumulator<T> union = builder.unionOf().nullType();
+ return addTypesToUnion(union, field.getChildren(), namespace);
+ } else {
+ Field headType = field.getChildren().get(0);
+ List<Field> tailTypes = field.getChildren().subList(1, field.getChildren().size());
+ SchemaBuilder.UnionAccumulator<T> union =
+ buildBaseTypeSchema(builder.unionOf(), headType, namespace);
+ return addTypesToUnion(union, tailTypes, namespace);
+ }
+ } else if (field.isNullable()) {
+ return buildBaseTypeSchema(builder.nullable(), field, namespace);
+ } else {
+ return buildBaseTypeSchema(builder, field, namespace);
+ }
+ }
+
+ private static <T> T buildBaseTypeSchema(
+ SchemaBuilder.BaseTypeBuilder<T> builder, Field field, String namespace) {
+
+ ArrowType.ArrowTypeID typeID = field.getType().getTypeID();
+
+ switch (typeID) {
+ case Null:
+ return builder.nullType();
+
+ case Bool:
+ return builder.booleanType();
+
+ case Int:
+ ArrowType.Int intType = (ArrowType.Int) field.getType();
+ if (intType.getBitWidth() > 32 || (intType.getBitWidth() == 32 && !intType.getIsSigned())) {
+ return builder.longType();
+ } else {
+ return builder.intType();
+ }
+
+ case FloatingPoint:
+ ArrowType.FloatingPoint floatType = (ArrowType.FloatingPoint) field.getType();
+ if (floatType.getPrecision() == FloatingPointPrecision.DOUBLE) {
+ return builder.doubleType();
+ } else {
+ return builder.floatType();
+ }
+
+ case Utf8:
+ return builder.stringType();
+
+ case Binary:
+ return builder.bytesType();
+
+ case FixedSizeBinary:
+ ArrowType.FixedSizeBinary fixedType = (ArrowType.FixedSizeBinary) field.getType();
+ String fixedTypeName = field.getName();
+ int fixedTypeWidth = fixedType.getByteWidth();
+ return builder.fixed(fixedTypeName).size(fixedTypeWidth);
+
+ case Decimal:
+ ArrowType.Decimal decimalType = (ArrowType.Decimal) field.getType();
+ return builder
+ .fixed(field.getName())
+ .prop("logicalType", "decimal")
+ .prop("precision", decimalType.getPrecision())
+ .prop("scale", decimalType.getScale())
+ .size(decimalType.getBitWidth() / 8);
+
+ case Date:
+ return builder.intBuilder().prop("logicalType", "date").endInt();
+
+ case Time:
+ ArrowType.Time timeType = (ArrowType.Time) field.getType();
+ if ((timeType.getUnit() == TimeUnit.SECOND || timeType.getUnit() == TimeUnit.MILLISECOND)) {
+ return builder.intBuilder().prop("logicalType", "time-millis").endInt();
+ } else {
+ // All other time types (sec, micro, nano) are encoded as time-micros (LONG)
+ return builder.longBuilder().prop("logicalType", "time-micros").endLong();
+ }
+
+ case Timestamp:
+ ArrowType.Timestamp timestampType = (ArrowType.Timestamp) field.getType();
+ String timestampLogicalType = timestampLogicalType(timestampType);
+ return builder.longBuilder().prop("logicalType", timestampLogicalType).endLong();
+
+ case Struct:
+ String childNamespace =
+ namespace == null ? field.getName() : namespace + "." + field.getName();
+ return buildRecordSchema(
+ builder.record(field.getName()), field.getChildren(), childNamespace);
+
+ case List:
+ case FixedSizeList:
+ return buildArraySchema(builder.array(), field, namespace);
+
+ case Map:
+ return buildMapSchema(builder.map(), field, namespace);
+
+ default:
+ throw new IllegalArgumentException(
+ "Element type not supported for Avro conversion: " + typeID.name());
+ }
+ }
+
+ private static <T> SchemaBuilder.FieldAssembler<T> buildBaseFieldSchema(
+ SchemaBuilder.BaseFieldTypeBuilder<T> builder, Field field, String namespace) {
+
+ ArrowType.ArrowTypeID typeID = field.getType().getTypeID();
+
+ switch (typeID) {
+ case Null:
+ return builder.nullType().noDefault();
+
+ case Bool:
+ return builder.booleanType().noDefault();
+
+ case Int:
+ ArrowType.Int intType = (ArrowType.Int) field.getType();
+ if (intType.getBitWidth() > 32 || (intType.getBitWidth() == 32 && !intType.getIsSigned())) {
+ return builder.longType().noDefault();
+ } else {
+ return builder.intType().noDefault();
+ }
+
+ case FloatingPoint:
+ ArrowType.FloatingPoint floatType = (ArrowType.FloatingPoint) field.getType();
+ if (floatType.getPrecision() == FloatingPointPrecision.DOUBLE) {
+ return builder.doubleType().noDefault();
+ } else {
+ return builder.floatType().noDefault();
+ }
+
+ case Utf8:
+ return builder.stringType().noDefault();
+
+ case Binary:
+ return builder.bytesType().noDefault();
+
+ case FixedSizeBinary:
+ ArrowType.FixedSizeBinary fixedType = (ArrowType.FixedSizeBinary) field.getType();
+ return builder.fixed(field.getName()).size(fixedType.getByteWidth()).noDefault();
+
+ case Decimal:
+ ArrowType.Decimal decimalType = (ArrowType.Decimal) field.getType();
+ return builder
+ .fixed(field.getName())
+ .prop("logicalType", "decimal")
+ .prop("precision", decimalType.getPrecision())
+ .prop("scale", decimalType.getScale())
+ .size(decimalType.getBitWidth() / 8)
+ .noDefault();
+
+ case Date:
+ return builder.intBuilder().prop("logicalType", "date").endInt().noDefault();
+
+ case Time:
+ ArrowType.Time timeType = (ArrowType.Time) field.getType();
+ if ((timeType.getUnit() == TimeUnit.SECOND || timeType.getUnit() == TimeUnit.MILLISECOND)) {
+ return builder.intBuilder().prop("logicalType", "time-millis").endInt().noDefault();
+ } else {
+ // All other time types (sec, micro, nano) are encoded as time-micros (LONG)
Review Comment:
nit: comment is out of sync with code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2685419869
Thanks so much for all the comments! I've put questions in line, will wait for your answers on how to address those points.
In the meantime I will add the schema generator and producers for the narrow primitive types, as well as some top level doc comments describing the translation. I'll make a start on the test coverage as well....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010632791
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroStructProducer.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer which produces nested record type values to avro encoder. Read the data from {@link
+ * org.apache.arrow.vector.complex.StructVector}.
+ */
+public class AvroStructProducer extends BaseAvroProducer<StructVector> {
+
+ private final Producer<? extends FieldVector>[] delegates;
+
+ /** Instantiate a AvroStructProducer. */
+ public AvroStructProducer(StructVector vector, Producer<? extends FieldVector>[] delegates) {
+ super(vector);
+ this.delegates = delegates;
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+
+ for (Producer<?> delegate : delegates) {
+ delegate.produce(encoder);
+ }
+
+ currentIndex++;
+ }
+
+ @Override
+ public void skipNull() {
+ for (Producer<?> delegate : delegates) {
+ delegate.skipNull();
+ }
+ super.skipNull();
+ }
+
+ @Override
+ public void setPosition(int index) {
+ for (Producer<?> delegate : delegates) {
+ delegate.setPosition(index);
+ }
+ super.setPosition(index);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public boolean resetValueVector(StructVector vector) {
+ for (int i = 0; i < delegates.length; i++) {
+ Producer<FieldVector> delegate = (Producer<FieldVector>) delegates[i];
+ delegate.resetValueVector(vector.getChildrenFromFields().get(i));
Review Comment:
Per above, nothing ever returns false and the value is never used, so I removed the return value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1972643154
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+import java.io.IOException;
+
+/**
+ * Interface that is used to produce values to avro encoder.
+ *
+ * @param <T> The vector within producer or its delegate, used for partially produce purpose.
+ */
+public interface Producer<T extends FieldVector> extends AutoCloseable {
+
+ /**
+ * Produce a specific type value from the vector and write it to avro encoder.
+ *
+ * @param encoder avro encoder to write data
+ * @throws IOException on error
+ */
+ void produce(Encoder encoder) throws IOException;
+
+ /** Skip null value in the vector by setting reader position + 1. */
+ void skipNull();
+
+ /** Set the position to read value from vector. */
+ void setPosition(int index);
Review Comment:
That sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1976672068
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+import java.io.IOException;
+
+/**
+ * Interface that is used to produce values to avro encoder.
+ *
+ * @param <T> The vector within producer or its delegate, used for partially produce purpose.
+ */
+public interface Producer<T extends FieldVector> extends AutoCloseable {
+
+ /**
+ * Produce a specific type value from the vector and write it to avro encoder.
+ *
+ * @param encoder avro encoder to write data
+ * @throws IOException on error
+ */
+ void produce(Encoder encoder) throws IOException;
+
+ /** Skip null value in the vector by setting reader position + 1. */
+ void skipNull();
+
+ /** Set the position to read value from vector. */
+ void setPosition(int index);
Review Comment:
Per my [comment below](https://github.com/apache/arrow-java/pull/638#issuecomment-2692815888), this proved more tricky than I first thought. In light of that, are you happy to stick with int for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2749052296
Hi @lidavidm , @wgtmac - Have pushed another update, here is a quick summary:
* I have been through all the comments above and responded. Please do come back on any you think need further attention!
* I have added tests for both schema and data production. Tests cover primitive types, logical types and containers of primitive and logical types. Some edges cases (max int values, UTF 8 text) are included.
* Tests do not include nested containers (e.g. list of lists), these may work but I've not tried it. Negative test cases are also not included (i.e. invalid / illegal / badly initialised data structures).
* Dictionary encoding not supported yet (we could push this to a second PR)
* Nullable types. I have [type, null] for nullable types instead of [null, type]. There was a comment on this. The current reader framework won't allow the latter, so we can't round-trip if we do it that way, but if you think it is preferable we could do it anyway now we have full coverage on the producers. Anyway I think a separate PR is needed to address issues in the consumers. Lmk what you'd like to do on this.
* Union data. Union schema production is working fine, but union data I really struggled with. Is there an issue with the field writers? I don't see how [this code](https://github.com/apache/arrow-java/blob/48a20ba91605ebb44df845671782acd8f869bd4b/vector/src/main/codegen/templates/DenseUnionWriter.java#L55) is ever going to work, since writers are not alll initialized. Also, per my understanding, the semantics of UnionWriter and DenseUnionWriter seem like they're the wrong way around. Fairly sure my understanding is the problem but I don't see where! Can you point me to any example code that populates a UnionVector and/or DenseUnionVector?
Apologies for the delay, lots else going on as usual but I think this is getting closer now. If you let me know what else you'd like to see for this to be considered ready, I'll try to turn it around as quickly as I can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2750660875
Thanks @lidavidm - good to know it is not just me going crazy! I have removed support of data production for unions for now. Also rebased on master.
Totally understand the time pressure. I might work on some of the other pieces in the meantime as separate PRs if that's OK, to have a couple of bits ready for you to look at :-)
Also read the thread on Unions. My company uses Arrow Java heavily in our product - time is a big issue for us as well but if I can find some I'll try to raise a few fixes that we're interested in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2750182221
I'm fine splitting things into other PRs if it makes your life easier.
It is known that the Java Union implementation is wrong, unfortunately. See https://github.com/apache/arrow-java/issues/108
If you want to punt on Union for now that's fine by me.
I'm a bit backlogged and will try to re-review when I get a chance but that may not be for a week or two, sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2750984663
Hi @lidavidm - There were two test failures in CI relating to nano precision time values that didn't show up in my local machine. I have pushed a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010644452
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroUint4Producer.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces long values from a {@link UInt4Vector}, writes data to an avro encoder.
+ */
+public class AvroUint4Producer extends BaseAvroProducer<UInt4Vector> {
+
+ /** Instantiate an AvroUint4Producer. */
+ public AvroUint4Producer(UInt4Vector vector) {
+ super(vector);
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ int unsigned = vector.getDataBuffer().getInt(currentIndex * (long) UInt4Vector.TYPE_WIDTH);
+ long signed = unsigned & 0xffffffffL;
Review Comment:
Thanks, updated the code to use that method.
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroUint2Producer.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.avro.io.Encoder;
+
+/** Producer that produces int values from a {@link UInt2Vector}, writes data to an avro encoder. */
+public class AvroUint2Producer extends BaseAvroProducer<UInt2Vector> {
+
+ /** Instantiate an AvroUint2Producer. */
+ public AvroUint2Producer(UInt2Vector vector) {
+ super(vector);
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ short unsigned = vector.getDataBuffer().getShort(currentIndex * (long) UInt2Vector.TYPE_WIDTH);
+ int signed = unsigned & 0xffff;
Review Comment:
Thanks, updated the code to use that method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2771358197
Ok well these CI failures aren't related to the PR, merging...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2017456583
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <table>
+ * <thead><tr><th>Arrow type</th><th>Avro encoding</th></tr></thead>
+ * <tbody>
+ * <tr><td>ArrowType.Null</td><td>NULL</td></tr>
+ * <tr><td>ArrowType.Bool</td><td>BOOLEAN</td></tr>
+ * <tr><td>ArrowType.Int(64 bit, unsigned 32 bit)</td><td>LONG</td></tr>
+ * <tr><td>ArrowType.Int(signed 32 bit, < 32 bit)</td><td>INT</td></tr>
+ * <tr><td>ArrowType.FloatingPoint(double)</td><td>DOUBLE</td></tr>
+ * <tr><td>ArrowType.FloatingPoint(single, half)</td><td>FLOAT</td></tr>
+ * <tr><td>ArrowType.Utf8</td><td>STRING</td></tr>
+ * <tr><td>ArrowType.LargeUtf8</td><td>STRING</td></tr>
+ * <tr><td>ArrowType.Binary</td><td>BYTES</td></tr>
+ * <tr><td>ArrowType.LargeBinary</td><td>BYTES</td></tr>
+ * <tr><td>ArrowType.FixedSizeBinary</td><td>FIXED</td></tr>
+ * <tr><td>ArrowType.Decimal</td><td>decimal (FIXED)</td></tr>
+ * <tr><td>ArrowType.Date</td><td>date (INT)</td></tr>
+ * <tr><td>ArrowType.Time (SEC | MILLI)</td><td>time-millis (INT)</td></tr>
+ * <tr><td>ArrowType.Time (MICRO | NANO)</td><td>time-micros (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (NANOSECONDS, TZ != NULL)</td><td>time-nanos (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MICROSECONDS, TZ != NULL)</td><td>time-micros (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ != NULL)</td><td>time-millis (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (NANOSECONDS, TZ == NULL)</td><td>local-time-nanos (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MICROSECONDS, TZ == NULL)</td><td>local-time-micros (LONG)</td></tr>
+ * <tr><td>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ == NULL)</td><td>local-time-millis (LONG)</td></tr>
+ * <tr><td>ArrowType.Duration</td><td>duration (FIXED)</td></tr>
+ * <tr><td>ArrowType.Interval</td><td>duration (FIXED)</td></tr>
+ * <tr><td>ArrowType.Struct</td><td>record</td></tr>
+ * <tr><td>ArrowType.List</td><td>array</td></tr>
+ * <tr><td>ArrowType.LargeList</td><td>array</td></tr>
+ * <tr><td>ArrowType.FixedSizeList</td><td>array</td></tr>
+ * <tr><td>ArrowType.Map</td><td>map</td></tr>
+ * <tr><td>ArrowType.Union</td><td>union</td></tr>
+ * </tbody>
+ * </table>
+ *
+ * <p>Nullable fields are represented as a union of [null | base-type]. Special treatment is given
+ * to nullability of unions - a union is considered nullable if the union field is nullable or any
+ * of its child fields are nullable. The schema for a nullable union will always contain a null
+ * type,none of the direct child types will be nullable.
+ *
+ * <p>List fields must contain precisely one child field, which may be nullable. Map fields are
+ * represented as a list of structs, where the struct fields are "key" and "value". The key field
+ * must always be of type STRING (Utf8) and cannot be nullable. The value can be of any type and
+ * may be nullable. Record types must contain at least one child field and cannot contain multiple
+ * fields with the same name
+ *
+ * @param arrowFields The arrow fields used to generate the Avro schema
+ * @param typeName Name of the top level Avro record type
+ * @param namespace Namespace of the top level Avro record type
+ * @return An Avro record schema for the given list of fields, with the specified name and
+ * namespace
+ */
+ public static Schema createAvroSchema(
+ List<Field> arrowFields, String typeName, String namespace) {
+ SchemaBuilder.RecordBuilder<Schema> assembler =
+ SchemaBuilder.record(typeName).namespace(namespace);
+ return buildRecordSchema(assembler, arrowFields, namespace);
+ }
+
+ /** Overload provided for convenience, sets namespace = null. */
+ public static Schema createAvroSchema(List<Field> arrowFields, String typeName) {
+ return createAvroSchema(arrowFields, typeName, null);
+ }
+
+ /** Overload provided for convenience, sets name = GENERIC_RECORD_TYPE_NAME. */
+ public static Schema createAvroSchema(List<Field> arrowFields) {
+ return createAvroSchema(arrowFields, GENERIC_RECORD_TYPE_NAME);
+ }
+
+ private static <T> T buildRecordSchema(
+ SchemaBuilder.RecordBuilder<T> builder, List<Field> fields, String namespace) {
+ if (fields.isEmpty()) {
+ throw new IllegalArgumentException("Record field must have at least one child field");
+ }
+ SchemaBuilder.FieldAssembler<T> assembler = builder.namespace(namespace).fields();
+ for (Field field : fields) {
+ assembler = buildFieldSchema(assembler, field, namespace);
+ }
+ return assembler.endRecord();
+ }
+
+ private static <T> SchemaBuilder.FieldAssembler<T> buildFieldSchema(
+ SchemaBuilder.FieldAssembler<T> assembler, Field field, String namespace) {
+
+ SchemaBuilder.FieldTypeBuilder<T> builder = assembler.name(field.getName()).type();
+
+ // Nullable unions need special handling, since union types cannot be directly nested
+ if (field.getType().getTypeID() == ArrowType.ArrowTypeID.Union) {
+ boolean unionNullable = field.getChildren().stream().anyMatch(Field::isNullable);
+ if (unionNullable) {
+ SchemaBuilder.UnionAccumulator<SchemaBuilder.NullDefault<T>> union =
+ builder.unionOf().nullType();
+ return addTypesToUnion(union, field.getChildren(), namespace).nullDefault();
+ } else {
+ Field headType = field.getChildren().get(0);
+ List<Field> tailTypes = field.getChildren().subList(1, field.getChildren().size());
+ SchemaBuilder.UnionAccumulator<SchemaBuilder.FieldDefault<T, ?>> union =
+ buildUnionFieldSchema(builder.unionOf(), headType, namespace);
+ return addTypesToUnion(union, tailTypes, namespace).noDefault();
+ }
+ } else if (field.isNullable()) {
+ return buildBaseFieldSchema(builder.nullable(), field, namespace);
+ } else {
+ return buildBaseFieldSchema(builder, field, namespace);
+ }
+ }
+
+ private static <T> T buildArraySchema(
+ SchemaBuilder.ArrayBuilder<T> builder, Field listField, String namespace) {
+ if (listField.getChildren().size() != 1) {
+ throw new IllegalArgumentException("List field must have exactly one child field");
+ }
+ Field itemField = listField.getChildren().get(0);
+ return buildTypeSchema(builder.items(), itemField, namespace);
+ }
+
+ private static <T> T buildMapSchema(
+ SchemaBuilder.MapBuilder<T> builder, Field mapField, String namespace) {
+ if (mapField.getChildren().size() != 1) {
+ throw new IllegalArgumentException("Map field must have exactly one child field");
+ }
+ Field entriesField = mapField.getChildren().get(0);
+ if (mapField.getChildren().size() != 1) {
+ throw new IllegalArgumentException("Map entries must have exactly two child fields");
+ }
+ Field keyField = entriesField.getChildren().get(0);
+ Field valueField = entriesField.getChildren().get(1);
+ if (keyField.getType().getTypeID() != ArrowType.ArrowTypeID.Utf8 || keyField.isNullable()) {
+ throw new IllegalArgumentException(
+ "Map keys must be of type string and cannot be nullable for conversion to Avro");
+ }
+ return buildTypeSchema(builder.values(), valueField, namespace);
+ }
+
+ private static <T> T buildTypeSchema(
+ SchemaBuilder.TypeBuilder<T> builder, Field field, String namespace) {
+
+ // Nullable unions need special handling, since union types cannot be directly nested
+ if (field.getType().getTypeID() == ArrowType.ArrowTypeID.Union) {
+ boolean unionNullable = field.getChildren().stream().anyMatch(Field::isNullable);
+ if (unionNullable) {
+ SchemaBuilder.UnionAccumulator<T> union = builder.unionOf().nullType();
+ return addTypesToUnion(union, field.getChildren(), namespace);
+ } else {
+ Field headType = field.getChildren().get(0);
+ List<Field> tailTypes = field.getChildren().subList(1, field.getChildren().size());
+ SchemaBuilder.UnionAccumulator<T> union =
+ buildBaseTypeSchema(builder.unionOf(), headType, namespace);
+ return addTypesToUnion(union, tailTypes, namespace);
+ }
+ } else if (field.isNullable()) {
+ return buildBaseTypeSchema(builder.nullable(), field, namespace);
+ } else {
+ return buildBaseTypeSchema(builder, field, namespace);
+ }
+ }
+
+ private static <T> T buildBaseTypeSchema(
+ SchemaBuilder.BaseTypeBuilder<T> builder, Field field, String namespace) {
+
+ ArrowType.ArrowTypeID typeID = field.getType().getTypeID();
+
+ switch (typeID) {
+ case Null:
+ return builder.nullType();
+
+ case Bool:
+ return builder.booleanType();
+
+ case Int:
+ ArrowType.Int intType = (ArrowType.Int) field.getType();
+ if (intType.getBitWidth() > 32 || (intType.getBitWidth() == 32 && !intType.getIsSigned())) {
+ return builder.longType();
+ } else {
+ return builder.intType();
+ }
+
+ case FloatingPoint:
+ ArrowType.FloatingPoint floatType = (ArrowType.FloatingPoint) field.getType();
+ if (floatType.getPrecision() == FloatingPointPrecision.DOUBLE) {
+ return builder.doubleType();
+ } else {
+ return builder.floatType();
+ }
+
+ case Utf8:
+ return builder.stringType();
+
+ case Binary:
+ return builder.bytesType();
+
+ case FixedSizeBinary:
+ ArrowType.FixedSizeBinary fixedType = (ArrowType.FixedSizeBinary) field.getType();
+ String fixedTypeName = field.getName();
+ int fixedTypeWidth = fixedType.getByteWidth();
+ return builder.fixed(fixedTypeName).size(fixedTypeWidth);
+
+ case Decimal:
+ ArrowType.Decimal decimalType = (ArrowType.Decimal) field.getType();
+ return builder
+ .fixed(field.getName())
+ .prop("logicalType", "decimal")
+ .prop("precision", decimalType.getPrecision())
+ .prop("scale", decimalType.getScale())
+ .size(decimalType.getBitWidth() / 8);
+
+ case Date:
+ return builder.intBuilder().prop("logicalType", "date").endInt();
+
+ case Time:
+ ArrowType.Time timeType = (ArrowType.Time) field.getType();
+ if ((timeType.getUnit() == TimeUnit.SECOND || timeType.getUnit() == TimeUnit.MILLISECOND)) {
+ return builder.intBuilder().prop("logicalType", "time-millis").endInt();
+ } else {
+ // All other time types (sec, micro, nano) are encoded as time-micros (LONG)
+ return builder.longBuilder().prop("logicalType", "time-micros").endLong();
+ }
+
+ case Timestamp:
+ ArrowType.Timestamp timestampType = (ArrowType.Timestamp) field.getType();
+ String timestampLogicalType = timestampLogicalType(timestampType);
+ return builder.longBuilder().prop("logicalType", timestampLogicalType).endLong();
+
+ case Struct:
+ String childNamespace =
+ namespace == null ? field.getName() : namespace + "." + field.getName();
+ return buildRecordSchema(
+ builder.record(field.getName()), field.getChildren(), childNamespace);
+
+ case List:
+ case FixedSizeList:
+ return buildArraySchema(builder.array(), field, namespace);
+
+ case Map:
+ return buildMapSchema(builder.map(), field, namespace);
+
+ default:
+ throw new IllegalArgumentException(
+ "Element type not supported for Avro conversion: " + typeID.name());
+ }
+ }
+
+ private static <T> SchemaBuilder.FieldAssembler<T> buildBaseFieldSchema(
+ SchemaBuilder.BaseFieldTypeBuilder<T> builder, Field field, String namespace) {
+
+ ArrowType.ArrowTypeID typeID = field.getType().getTypeID();
+
+ switch (typeID) {
+ case Null:
+ return builder.nullType().noDefault();
+
+ case Bool:
+ return builder.booleanType().noDefault();
+
+ case Int:
+ ArrowType.Int intType = (ArrowType.Int) field.getType();
+ if (intType.getBitWidth() > 32 || (intType.getBitWidth() == 32 && !intType.getIsSigned())) {
+ return builder.longType().noDefault();
+ } else {
+ return builder.intType().noDefault();
+ }
+
+ case FloatingPoint:
+ ArrowType.FloatingPoint floatType = (ArrowType.FloatingPoint) field.getType();
+ if (floatType.getPrecision() == FloatingPointPrecision.DOUBLE) {
+ return builder.doubleType().noDefault();
+ } else {
+ return builder.floatType().noDefault();
+ }
+
+ case Utf8:
+ return builder.stringType().noDefault();
+
+ case Binary:
+ return builder.bytesType().noDefault();
+
+ case FixedSizeBinary:
+ ArrowType.FixedSizeBinary fixedType = (ArrowType.FixedSizeBinary) field.getType();
+ return builder.fixed(field.getName()).size(fixedType.getByteWidth()).noDefault();
+
+ case Decimal:
+ ArrowType.Decimal decimalType = (ArrowType.Decimal) field.getType();
+ return builder
+ .fixed(field.getName())
+ .prop("logicalType", "decimal")
+ .prop("precision", decimalType.getPrecision())
+ .prop("scale", decimalType.getScale())
+ .size(decimalType.getBitWidth() / 8)
+ .noDefault();
+
+ case Date:
+ return builder.intBuilder().prop("logicalType", "date").endInt().noDefault();
+
+ case Time:
+ ArrowType.Time timeType = (ArrowType.Time) field.getType();
+ if ((timeType.getUnit() == TimeUnit.SECOND || timeType.getUnit() == TimeUnit.MILLISECOND)) {
+ return builder.intBuilder().prop("logicalType", "time-millis").endInt().noDefault();
+ } else {
+ // All other time types (sec, micro, nano) are encoded as time-micros (LONG)
Review Comment:
Comments updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1972643290
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/BaseAvroProducer.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+
+/**
+ * Base class for avro producers.
+ *
+ * @param <T> vector type.
+ */
+public abstract class BaseAvroProducer<T extends FieldVector> implements Producer<T> {
+
+ protected T vector;
+ protected int currentIndex;
+
+ /**
+ * Constructs a base avro consumer.
+ *
+ * @param vector the vector to consume.
+ */
+ protected BaseAvroProducer(T vector) {
+ this.vector = vector;
+ }
+
+ @Override
+ public void skipNull() {
+ currentIndex++;
+ }
+
+ @Override
+ public void setPosition(int index) {
+ currentIndex = index;
+ }
+
+ @Override
+ public void close() throws Exception {
+ vector.close();
+ }
+
+ @Override
+ public boolean resetValueVector(T vector) {
+ this.vector = vector;
+ this.currentIndex = 0;
+ return true;
+ }
Review Comment:
Yeah, let's drop `close()` and `AutoClosable`!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1974675294
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroDecimalProducer.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces decimal values from a {@link DecimalVector}, writes data to an Avro
+ * encoder.
+ */
+public abstract class AvroDecimalProducer extends BaseAvroProducer<DecimalVector> {
+
+ /** Instantiate an AvroDecimalProducer. */
+ protected AvroDecimalProducer(DecimalVector vector) {
+ super(vector);
+ }
+
+ /** Producer for decimal logical type with original bytes type. */
+ public static class BytesDecimalProducer extends AvroDecimalProducer {
+
+ private final byte[] reuseBytes;
+
+ /** Instantiate a BytesDecimalConsumer. */
+ public BytesDecimalProducer(DecimalVector vector) {
+ super(vector);
+ Preconditions.checkArgument(
+ vector.getTypeWidth() <= 16, "Decimal bytes length should <= 16.");
+ reuseBytes = new byte[vector.getTypeWidth()];
Review Comment:
Is this correct? I don't think the type width is constant for each value. It might be similar to what you did in `AvroBytesProducer`.
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/BaseAvroProducer.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+
+/**
+ * Base class for avro producers.
+ *
+ * @param <T> vector type.
+ */
+public abstract class BaseAvroProducer<T extends FieldVector> implements Producer<T> {
+
+ protected T vector;
+ protected int currentIndex;
+
+ /**
+ * Constructs a base avro consumer.
+ *
+ * @param vector the vector to consume.
+ */
+ protected BaseAvroProducer(T vector) {
+ this.vector = vector;
+ }
+
+ @Override
+ public void skipNull() {
+ currentIndex++;
+ }
+
+ @Override
+ public void setPosition(int index) {
+ currentIndex = index;
Review Comment:
Is there any restriction to the new `index`? Do we expect `index` < `currentIndex`?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroArraysProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDoubleProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloatProducer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroLongProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionsProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMillisProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMillisProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.Types;
+
+public class ArrowToAvroUtils {
+
+ /**
+ * Create a composite Avro producer for a set of field vectors (typically the root set of a VSR).
+ *
+ * @param vectors The vectors that will be used to produce Avro data
+ * @return The resulting composite Avro producer
+ */
+ public static CompositeAvroProducer createCompositeProducer(List<FieldVector> vectors) {
+
+ List<Producer<? extends FieldVector>> producers = new ArrayList<>(vectors.size());
+
+ for (FieldVector vector : vectors) {
+ BaseAvroProducer<? extends FieldVector> producer = createProducer(vector);
+ producers.add(producer);
+ }
+
+ return new CompositeAvroProducer(producers);
+ }
+
+ private static BaseAvroProducer<?> createProducer(FieldVector vector) {
+ boolean nullable = vector.getField().isNullable();
+ return createProducer(vector, nullable);
+ }
+
+ private static BaseAvroProducer<?> createProducer(FieldVector vector, boolean nullable) {
+
+ Preconditions.checkNotNull(vector, "Arrow vector object can't be null");
+
+ final Types.MinorType minorType = vector.getMinorType();
+
+ // Avro understands nullable types as a union of type | null
+ // Most nullable fields in a VSR will not be unions, so provide a special wrapper
+ if (nullable && minorType != Types.MinorType.UNION) {
+ final BaseAvroProducer<?> innerProducer = createProducer(vector, false);
+ return new AvroNullableProducer<>(innerProducer);
+ }
+
+ switch (minorType) {
+
+ // Primitive types with direct mapping to Avro
+
+ case NULL:
+ return new AvroNullProducer((NullVector) vector);
+ case BIT:
+ return new AvroBooleanProducer((BitVector) vector);
+ case INT:
+ return new AvroIntProducer((IntVector) vector);
+ case BIGINT:
+ return new AvroLongProducer((BigIntVector) vector);
+ case FLOAT4:
+ return new AvroFloatProducer((Float4Vector) vector);
+ case FLOAT8:
+ return new AvroDoubleProducer((Float8Vector) vector);
+ case VARBINARY:
+ return new AvroBytesProducer((VarBinaryVector) vector);
+ case FIXEDSIZEBINARY:
+ return new AvroFixedProducer((FixedSizeBinaryVector) vector);
+ case VARCHAR:
+ return new AvroStringProducer((VarCharVector) vector);
+
+ // Logical types
+
+ case DECIMAL:
+ return new AvroDecimalProducer.FixedDecimalProducer(
+ (DecimalVector) vector, DecimalVector.TYPE_WIDTH);
+ case DATEDAY:
+ return new AvroDateProducer((DateDayVector) vector);
+ case TIMEMILLI:
+ return new AvroTimeMillisProducer((TimeMilliVector) vector);
+ case TIMEMICRO:
+ return new AvroTimeMicroProducer((TimeMicroVector) vector);
+ case TIMESTAMPMILLI:
+ return new AvroTimestampMillisProducer((TimeStampMilliVector) vector);
+ case TIMESTAMPMICRO:
+ return new AvroTimestampMicroProducer((TimeStampMicroVector) vector);
+
+ // Complex types
+
+ case STRUCT:
+ StructVector structVector = (StructVector) vector;
+ List<FieldVector> childVectors = structVector.getChildrenFromFields();
+ Producer<?>[] childProducers = new Producer<?>[childVectors.size()];
+ for (int i = 0; i < childVectors.size(); i++) {
+ FieldVector childVector = childVectors.get(i);
+ childProducers[i] = createProducer(childVector, childVector.getField().isNullable());
+ }
+ return new AvroStructProducer(structVector, childProducers);
+
+ case LIST:
+ ListVector listVector = (ListVector) vector;
+ FieldVector itemVector = listVector.getDataVector();
+ Producer<?> itemProducer = createProducer(itemVector, itemVector.getField().isNullable());
+ return new AvroArraysProducer(listVector, itemProducer);
+
+ case MAP:
+ MapVector mapVector = (MapVector) vector;
+ StructVector entryVector = (StructVector) mapVector.getDataVector();
+ VarCharVector keyVector = (VarCharVector) entryVector.getChildrenFromFields().get(0);
Review Comment:
What if the key is not a string type?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroNullableProducer.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer wrapper which producers nullable types to an avro encoder. Write the data to the
+ * underlying {@link FieldVector}.
+ */
+public class AvroNullableProducer<T extends FieldVector> extends BaseAvroProducer<T> {
+
+ private final Producer<T> delegate;
+
+ /** Instantiate a AvroNullableProducer. */
+ public AvroNullableProducer(Producer<T> delegate) {
+ super(delegate.getVector());
+ this.delegate = delegate;
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ if (vector.isNull(currentIndex)) {
+ encoder.writeInt(1);
Review Comment:
Does it mean that you placed `[type, null]` as the union type? Is it a good convention to put `null` before `type` so the default value is null when reader type does not match the writer type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010646441
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDenseUnionProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.DenseUnionVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <ul>
+ * <li>ArrowType.Null --> NULL
+ * <li>ArrowType.Bool --> BOOLEAN
+ * <li>ArrowType.Int(64 bit, unsigned 32 bit) --> LONG
+ * <li>ArrowType.Int(signed 32 bit, < 32 bit) --> INT
+ * <li>ArrowType.FloatingPoint(double) --> DOUBLE
+ * <li>ArrowType.FloatingPoint(single, half) --> FLOAT
+ * <li>ArrowType.Utf8 --> STRING
+ * <li>ArrowType.LargeUtf8 --> STRING
+ * <li>ArrowType.Binary --> BYTES
+ * <li>ArrowType.LargeBinary --> BYTES
+ * <li>ArrowType.FixedSizeBinary --> FIXED
+ * <li>ArrowType.Decimal --> decimal (FIXED)
+ * <li>ArrowType.Date --> date (INT)
+ * <li>ArrowType.Time (MILLI) --> time-millis (INT)
+ * <li>ArrowType.Time (SEC | MICRO | NANO) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ != NULL) --> time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ != NULL) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ != NULL) --> time-millis (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ == NULL) --> local-time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ == NULL) --> local-time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ == NULL) --> local-time-millis (LONG)
+ * <li>ArrowType.Duration --> duration (FIXED)
+ * <li>ArrowType.Interval --> duration (FIXED)
+ * <li>ArrowType.Struct --> record
+ * <li>ArrowType.List --> array
+ * <li>ArrowType.LargeList --> array
+ * <li>ArrowType.FixedSizeList --> array
+ * <li>ArrowType.Map --> map
+ * <li>ArrowType.Union --> union
+ * </ul>
+ *
+ * <p>Nullable fields are represented as a union of [null | base-type]. Special treatment is given
+ * to nullability of unions - a union is considered nullable if the union field is nullable or any
Review Comment:
I checked the spec and yes you are correct! I have updated the logic accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010645566
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDenseUnionProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.DenseUnionVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <ul>
+ * <li>ArrowType.Null --> NULL
Review Comment:
I've converted the type mapping info into and HTML table - shows up nicely in the IDE auto complete :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2702107186
Thanks @lidavidm for all the comments - really helpful :-) I'll push an update in a few days to address these and also add the tests, will probably be after the weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2759141103
Hi - I think I have addressed all the new comments. Also found an issue in the fixed list producer - I updated the test cases to show this up and included a fix. Fairly sure all the complex type producers are correct now but would be worth a set of eyes!! The four I have are list, fixed size list, map and struct.
Please let me know if there is anything else needs addressing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2770756352
This last update uses Avro's logical type classes instead of setting the raw schema props. The output to disk is the same, but if you want to use the generated Avro Schema directly these structures should be available. Needed for round-trip testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2017456018
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroListProducer.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces array values from a {@link ListVector}, writes data to an avro encoder.
+ */
+public class AvroListProducer extends BaseAvroProducer<ListVector> {
+
+ private final Producer<? extends FieldVector> delegate;
+
+ /** Instantiate an AvroListProducer. */
+ public AvroListProducer(ListVector vector, Producer<? extends FieldVector> delegate) {
+ super(vector);
+ this.delegate = delegate;
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+
+ int startOffset = vector.getOffsetBuffer().getInt(currentIndex * (long) Integer.BYTES);
+ int endOffset = vector.getOffsetBuffer().getInt((currentIndex + 1) * (long) Integer.BYTES);
+ int nItems = endOffset - startOffset;
+
+ encoder.writeArrayStart();
+ encoder.setItemCount(nItems);
+
+ for (int i = 0; i < nItems; i++) {
+ encoder.startItem();
+ delegate.produce(encoder);
+ }
+
+ encoder.writeArrayEnd();
+ currentIndex++;
+ }
+
+ // Do not override skipNull(), delegate will not have an entry if the list is null
+
+ @Override
+ public void setPosition(int index) {
+ if (index < 0 || index > vector.getValueCount()) {
Review Comment:
I have added a doc comment on the base class as you suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2699709876
Sorry, I've gotten caught up in my main job - I'll try to circle back here soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2683793488
<!-- label_helper_comment -->
Thank you for opening a pull request!
Please label the PR with one or more of:
- bug-fix
- chore
- dependencies
- documentation
- enhancement
Also, add the 'breaking-change' label if appropriate.
See [CONTRIBUTING.md](https://github.com/apache/arrow-java/blob/main/CONTRIBUTING.md) for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1971822027
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/CompositeAvroProducer.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.avro.io.Encoder;
+
+/** Composite producer which holds all producers. It manages the produce and cleanup process. */
+public class CompositeAvroProducer implements AutoCloseable {
+
+ private final List<Producer<? extends FieldVector>> producers;
+
+ public CompositeAvroProducer(List<Producer<? extends FieldVector>> producers) {
+ this.producers = producers;
+ }
+
+ public List<Producer<?>> getProducers() {
+ return producers;
+ }
+
+ /** Produce encoder data. */
+ public void produce(Encoder encoder) throws IOException {
+ for (Producer<? extends FieldVector> producer : producers) {
+ producer.produce(encoder);
+ }
+ }
+
+ /** Reset vector of consumers with the given {@link VectorSchemaRoot}. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public void resetProducerVectors(VectorSchemaRoot root) {
+ // This method assumes that the VSR matches the constructed set of producers
+ int index = 0;
+ for (Producer producer : producers) {
+ if (producer.resetValueVector(root.getFieldVectors().get(index))) {
+ index++;
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ // clean up
+ try {
+ AutoCloseables.close(producers);
+ } catch (Exception e) {
+ throw new RuntimeException("Error occurs in close.", e);
Review Comment:
Per above, if producers are not owning / closing the vectors this issue goes away. Otherwise yes this would make sense. Let me know, which you'd like me to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010642572
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroTimeMilliProducer.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.vector.TimeMilliVector;
+
+/**
+ * Producer that produces time (milliseconds) values from a {@link TimeMilliVector}, writes data to
+ * an Avro encoder.
+ */
+public class AvroTimeMilliProducer extends AvroIntProducer {
+
+ // Time in milliseconds stored as integer, matches Avro time-millis type
Review Comment:
I think time for seconds / millis are four byte width? [TimeMilliVector](https://github.com/apache/arrow-java/blob/48a20ba91605ebb44df845671782acd8f869bd4b/vector/src/main/java/org/apache/arrow/vector/TimeMilliVector.java#L39). So shouldn't be a range issue for this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1976673041
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroArraysProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDoubleProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloatProducer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroLongProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionsProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMillisProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMillisProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.Types;
+
+public class ArrowToAvroUtils {
+
+ /**
+ * Create a composite Avro producer for a set of field vectors (typically the root set of a VSR).
+ *
+ * @param vectors The vectors that will be used to produce Avro data
+ * @return The resulting composite Avro producer
+ */
+ public static CompositeAvroProducer createCompositeProducer(List<FieldVector> vectors) {
+
+ List<Producer<? extends FieldVector>> producers = new ArrayList<>(vectors.size());
+
+ for (FieldVector vector : vectors) {
+ BaseAvroProducer<? extends FieldVector> producer = createProducer(vector);
+ producers.add(producer);
+ }
+
+ return new CompositeAvroProducer(producers);
+ }
+
+ private static BaseAvroProducer<?> createProducer(FieldVector vector) {
+ boolean nullable = vector.getField().isNullable();
+ return createProducer(vector, nullable);
+ }
+
+ private static BaseAvroProducer<?> createProducer(FieldVector vector, boolean nullable) {
+
+ Preconditions.checkNotNull(vector, "Arrow vector object can't be null");
+
+ final Types.MinorType minorType = vector.getMinorType();
+
+ // Avro understands nullable types as a union of type | null
+ // Most nullable fields in a VSR will not be unions, so provide a special wrapper
+ if (nullable && minorType != Types.MinorType.UNION) {
+ final BaseAvroProducer<?> innerProducer = createProducer(vector, false);
+ return new AvroNullableProducer<>(innerProducer);
+ }
+
+ switch (minorType) {
+
+ // Primitive types with direct mapping to Avro
+
+ case NULL:
+ return new AvroNullProducer((NullVector) vector);
+ case BIT:
+ return new AvroBooleanProducer((BitVector) vector);
+ case INT:
+ return new AvroIntProducer((IntVector) vector);
+ case BIGINT:
+ return new AvroLongProducer((BigIntVector) vector);
+ case FLOAT4:
+ return new AvroFloatProducer((Float4Vector) vector);
+ case FLOAT8:
+ return new AvroDoubleProducer((Float8Vector) vector);
+ case VARBINARY:
+ return new AvroBytesProducer((VarBinaryVector) vector);
+ case FIXEDSIZEBINARY:
+ return new AvroFixedProducer((FixedSizeBinaryVector) vector);
+ case VARCHAR:
+ return new AvroStringProducer((VarCharVector) vector);
+
+ // Logical types
+
+ case DECIMAL:
+ return new AvroDecimalProducer.FixedDecimalProducer(
+ (DecimalVector) vector, DecimalVector.TYPE_WIDTH);
+ case DATEDAY:
+ return new AvroDateProducer((DateDayVector) vector);
+ case TIMEMILLI:
+ return new AvroTimeMillisProducer((TimeMilliVector) vector);
+ case TIMEMICRO:
+ return new AvroTimeMicroProducer((TimeMicroVector) vector);
+ case TIMESTAMPMILLI:
+ return new AvroTimestampMillisProducer((TimeStampMilliVector) vector);
+ case TIMESTAMPMICRO:
+ return new AvroTimestampMicroProducer((TimeStampMicroVector) vector);
+
+ // Complex types
+
+ case STRUCT:
+ StructVector structVector = (StructVector) vector;
+ List<FieldVector> childVectors = structVector.getChildrenFromFields();
+ Producer<?>[] childProducers = new Producer<?>[childVectors.size()];
+ for (int i = 0; i < childVectors.size(); i++) {
+ FieldVector childVector = childVectors.get(i);
+ childProducers[i] = createProducer(childVector, childVector.getField().isNullable());
+ }
+ return new AvroStructProducer(structVector, childProducers);
+
+ case LIST:
+ ListVector listVector = (ListVector) vector;
+ FieldVector itemVector = listVector.getDataVector();
+ Producer<?> itemProducer = createProducer(itemVector, itemVector.getField().isNullable());
+ return new AvroArraysProducer(listVector, itemProducer);
+
+ case MAP:
+ MapVector mapVector = (MapVector) vector;
+ StructVector entryVector = (StructVector) mapVector.getDataVector();
+ VarCharVector keyVector = (VarCharVector) entryVector.getChildrenFromFields().get(0);
Review Comment:
Good point! I put a check in the schema generation but not here. In Avro keys have to be string so if that is not the case I have thrown an error. Please leave this open and I will fix - out of time for today!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2692815888
Hello. I have pushed a second iteration for review. Major updates are:
* Added schema generation, to convert Arrow List<Field> to Avro Schema (record schema). I added doc comments with the type mapping. Schema is built of ArrowType while physical producers use MinorType, that bugged me at first but on reflection probably seems correct?
* Updated the producers with a more complete set of types, including all widths of float / int types and logical types for all the date / time types (but not duration as of yet). Would appreciate a set of eyes on the timezone mapping for TZ aware timestamps! Per my reading of the Avro spec this is correct but worth double checking.
I have tried to make it so most Arrow types will map to an Avro type if there is a suitable target. The basic approach is:
* If a direct equivalent exists, use that
* Next, choose an equivalent with no loss of precision (e.g. small int -> int)
* Next, choose an equivalent with loss of precision but no loss of range (e.g. time nano -> time micros)
* Finally, choose an equivalent with the best match and error on overflow (e.g. unsigned long -> long)
I'm not sure about the last one, would it be better just to say we can't convert those types and throw an error up front?
Large types (for text, binary, list) are not supported yet, there will be some extra thought needed on getting very large buffers between the two APIs. Interval / duration also not done yet, Arrow Interval resembles Avro Duration but is not an exact match and Arrow duration doesn't really have an equivalent that I could see. View types also not included as of now.
I tried to move over to using long as the current row index, however a lot of the high-level APIs on the vector types use ints. For working directly with primitive types I was able to go to the underlying buffer (there were reasons to do that anyway in a few cases), but for the complex types that gets trickier because the type of the child vectors is not known. Is it ok to keep the interface using an int for current index for now? Most of the primitive producers are updated anyway, so the work to move over later would not be too much.
I've run out of time for today. Next I plan to do the tests - like the consumers, will be exhaustive across types, combinations of nullability etc with some edge cases. Then the last thing I'd like to come back to on this PR is enums / dict encoding, just in case that has any impact on the structure of things would be good to know while it's still easy to change things.
I have addressed some of the comments but not all, I will work through the rest but wanted to share this to check I'm still going in the right direction!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1976672861
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroNullableProducer.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer wrapper which producers nullable types to an avro encoder. Write the data to the
+ * underlying {@link FieldVector}.
+ */
+public class AvroNullableProducer<T extends FieldVector> extends BaseAvroProducer<T> {
+
+ private final Producer<T> delegate;
+
+ /** Instantiate a AvroNullableProducer. */
+ public AvroNullableProducer(Producer<T> delegate) {
+ super(delegate.getVector());
+ this.delegate = delegate;
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ if (vector.isNull(currentIndex)) {
+ encoder.writeInt(1);
Review Comment:
I tried doing that, it broke the consumers and the data would not do round trip. In Avro this is considered as the difference between "nullable" and "optional", with "optional" being the semantics you are suggesting. I have kept it as nullable for now. I do think we will need to follow up with a PR on the consumers, particularly to address null handling. Nullable types get mangled in the schema round trip at the moment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1974765389
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroDecimalProducer.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces decimal values from a {@link DecimalVector}, writes data to an Avro
+ * encoder.
+ */
+public abstract class AvroDecimalProducer extends BaseAvroProducer<DecimalVector> {
+
+ /** Instantiate an AvroDecimalProducer. */
+ protected AvroDecimalProducer(DecimalVector vector) {
+ super(vector);
+ }
+
+ /** Producer for decimal logical type with original bytes type. */
+ public static class BytesDecimalProducer extends AvroDecimalProducer {
+
+ private final byte[] reuseBytes;
+
+ /** Instantiate a BytesDecimalConsumer. */
+ public BytesDecimalProducer(DecimalVector vector) {
+ super(vector);
+ Preconditions.checkArgument(
+ vector.getTypeWidth() <= 16, "Decimal bytes length should <= 16.");
+ reuseBytes = new byte[vector.getTypeWidth()];
Review Comment:
Well, I think I have mixed up the requirement from Iceberg which says `Stored as fixed using the minimum number of bytes for the given precision.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2017458056
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDenseUnionProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.DenseUnionVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <ul>
+ * <li>ArrowType.Null --> NULL
+ * <li>ArrowType.Bool --> BOOLEAN
+ * <li>ArrowType.Int(64 bit, unsigned 32 bit) --> LONG
+ * <li>ArrowType.Int(signed 32 bit, < 32 bit) --> INT
+ * <li>ArrowType.FloatingPoint(double) --> DOUBLE
+ * <li>ArrowType.FloatingPoint(single, half) --> FLOAT
+ * <li>ArrowType.Utf8 --> STRING
+ * <li>ArrowType.LargeUtf8 --> STRING
+ * <li>ArrowType.Binary --> BYTES
+ * <li>ArrowType.LargeBinary --> BYTES
+ * <li>ArrowType.FixedSizeBinary --> FIXED
+ * <li>ArrowType.Decimal --> decimal (FIXED)
+ * <li>ArrowType.Date --> date (INT)
+ * <li>ArrowType.Time (MILLI) --> time-millis (INT)
+ * <li>ArrowType.Time (SEC | MICRO | NANO) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ != NULL) --> time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ != NULL) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ != NULL) --> time-millis (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ == NULL) --> local-time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ == NULL) --> local-time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ == NULL) --> local-time-millis (LONG)
+ * <li>ArrowType.Duration --> duration (FIXED)
+ * <li>ArrowType.Interval --> duration (FIXED)
+ * <li>ArrowType.Struct --> record
+ * <li>ArrowType.List --> array
+ * <li>ArrowType.LargeList --> array
+ * <li>ArrowType.FixedSizeList --> array
+ * <li>ArrowType.Map --> map
+ * <li>ArrowType.Union --> union
+ * </ul>
+ *
+ * <p>Nullable fields are represented as a union of [null | base-type]. Special treatment is given
+ * to nullability of unions - a union is considered nullable if the union field is nullable or any
Review Comment:
I have updated the comment on unions. Also updated the comment for nullable types, which are encoded as [ base-type | null ], per the discussion above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1971812587
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
Review Comment:
I'm following the naming pattern from the consumers, the top level interface there is just `Consumer`. I've matched the type names for all the individual type producers as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010630423
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Interface that is used to produce values to avro encoder.
+ *
+ * @param <T> The vector within producer or its delegate, used for partially produce purpose.
+ */
+public interface Producer<T extends FieldVector> {
+
+ /**
+ * Produce a specific type value from the vector and write it to avro encoder.
+ *
+ * @param encoder avro encoder to write data
+ * @throws IOException on error
+ */
+ void produce(Encoder encoder) throws IOException;
+
+ /** Skip null value in the vector by setting reader position + 1. */
+ void skipNull();
+
+ /** Set the position to read value from vector. */
+ void setPosition(int index);
+
+ /**
+ * Reset the vector within producer.
+ *
+ * @return true if reset is successful, false if reset is not needed.
+ */
+ boolean resetValueVector(T vector);
Review Comment:
Nothing ever returns false and the value is never used. I have changed this to return void.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2751468881
CI looks good now - just the link missing on the PR to issue #615, sorry I don't have permission to add that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2770501896
Hello - Thought I would make a start on step 2 in the Avro series. I have raised #698 to describe what I'm thinking.
Still very happy to do another round of updates on this one if there are more comments as / when people have time to review - lmk :-) But since there are a few more PRs in the series to get the full capability I thought it was worth starting on the second one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2770979645
The proposal looks good to me. Sorry, I will try to give a hopefully final review here this week - as you have seen I tend to be scattered about so I appreciate your patience
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1976672431
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroDecimalProducer.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces decimal values from a {@link DecimalVector}, writes data to an Avro
+ * encoder.
+ */
+public abstract class AvroDecimalProducer extends BaseAvroProducer<DecimalVector> {
+
+ /** Instantiate an AvroDecimalProducer. */
+ protected AvroDecimalProducer(DecimalVector vector) {
+ super(vector);
+ }
+
+ /** Producer for decimal logical type with original bytes type. */
+ public static class BytesDecimalProducer extends AvroDecimalProducer {
+
+ private final byte[] reuseBytes;
+
+ /** Instantiate a BytesDecimalConsumer. */
+ public BytesDecimalProducer(DecimalVector vector) {
+ super(vector);
+ Preconditions.checkArgument(
+ vector.getTypeWidth() <= 16, "Decimal bytes length should <= 16.");
+ reuseBytes = new byte[vector.getTypeWidth()];
Review Comment:
Apologies, have removed this, I was mirroring consumers which need to handle both bytes and fixed inputs, but for producers we can always used fixed and the type width is always known.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1976671839
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/BaseAvroProducer.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+
+/**
+ * Base class for avro producers.
+ *
+ * @param <T> vector type.
+ */
+public abstract class BaseAvroProducer<T extends FieldVector> implements Producer<T> {
+
+ protected T vector;
+ protected int currentIndex;
+
+ /**
+ * Constructs a base avro consumer.
+ *
+ * @param vector the vector to consume.
+ */
+ protected BaseAvroProducer(T vector) {
+ this.vector = vector;
+ }
+
+ @Override
+ public void skipNull() {
+ currentIndex++;
+ }
+
+ @Override
+ public void setPosition(int index) {
+ currentIndex = index;
+ }
+
+ @Override
+ public void close() throws Exception {
+ vector.close();
+ }
+
+ @Override
+ public boolean resetValueVector(T vector) {
+ this.vector = vector;
+ this.currentIndex = 0;
+ return true;
+ }
Review Comment:
Done, close() and AutoClosable are gone :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1970974511
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
Review Comment:
nit: AvroProducer since everything else is named like that?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/CompositeAvroProducer.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.avro.io.Encoder;
+
+/** Composite producer which holds all producers. It manages the produce and cleanup process. */
+public class CompositeAvroProducer implements AutoCloseable {
+
+ private final List<Producer<? extends FieldVector>> producers;
+
+ public CompositeAvroProducer(List<Producer<? extends FieldVector>> producers) {
+ this.producers = producers;
+ }
+
+ public List<Producer<?>> getProducers() {
+ return producers;
+ }
+
+ /** Produce encoder data. */
+ public void produce(Encoder encoder) throws IOException {
+ for (Producer<? extends FieldVector> producer : producers) {
+ producer.produce(encoder);
+ }
+ }
+
+ /** Reset vector of consumers with the given {@link VectorSchemaRoot}. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public void resetProducerVectors(VectorSchemaRoot root) {
+ // This method assumes that the VSR matches the constructed set of producers
+ int index = 0;
+ for (Producer producer : producers) {
+ if (producer.resetValueVector(root.getFieldVectors().get(index))) {
+ index++;
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ // clean up
+ try {
+ AutoCloseables.close(producers);
+ } catch (Exception e) {
+ throw new RuntimeException("Error occurs in close.", e);
Review Comment:
Normally `close` is declared to throw `Exception` anyways - do we need to rewrap this in RuntimeException?
Otherwise use `AutoCloseables.closeNoChecked`
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+import java.io.IOException;
+
+/**
+ * Interface that is used to produce values to avro encoder.
+ *
+ * @param <T> The vector within producer or its delegate, used for partially produce purpose.
+ */
+public interface Producer<T extends FieldVector> extends AutoCloseable {
+
+ /**
+ * Produce a specific type value from the vector and write it to avro encoder.
+ *
+ * @param encoder avro encoder to write data
+ * @throws IOException on error
+ */
+ void produce(Encoder encoder) throws IOException;
+
+ /** Skip null value in the vector by setting reader position + 1. */
+ void skipNull();
+
+ /** Set the position to read value from vector. */
+ void setPosition(int index);
Review Comment:
I think we've been trying to be future-proof and use `long` for indices now (since eventually once we support the new memory APIs we'll be able to address more than 2GiB of memory at once)
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/BaseAvroProducer.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+
+/**
+ * Base class for avro producers.
+ *
+ * @param <T> vector type.
+ */
+public abstract class BaseAvroProducer<T extends FieldVector> implements Producer<T> {
+
+ protected T vector;
+ protected int currentIndex;
+
+ /**
+ * Constructs a base avro consumer.
+ *
+ * @param vector the vector to consume.
+ */
+ protected BaseAvroProducer(T vector) {
+ this.vector = vector;
+ }
+
+ @Override
+ public void skipNull() {
+ currentIndex++;
+ }
+
+ @Override
+ public void setPosition(int index) {
+ currentIndex = index;
+ }
+
+ @Override
+ public void close() throws Exception {
+ vector.close();
+ }
+
+ @Override
+ public boolean resetValueVector(T vector) {
+ this.vector = vector;
+ this.currentIndex = 0;
+ return true;
+ }
Review Comment:
Hmm. Should resetting close the prior vector? If not, then maybe we also shouldn't be closing the vector here (i.e. it's the upper level code's responsibility to manage the vector lifetime and the producer is just temporarily holding the vector, not owning it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1971819669
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/BaseAvroProducer.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+
+/**
+ * Base class for avro producers.
+ *
+ * @param <T> vector type.
+ */
+public abstract class BaseAvroProducer<T extends FieldVector> implements Producer<T> {
+
+ protected T vector;
+ protected int currentIndex;
+
+ /**
+ * Constructs a base avro consumer.
+ *
+ * @param vector the vector to consume.
+ */
+ protected BaseAvroProducer(T vector) {
+ this.vector = vector;
+ }
+
+ @Override
+ public void skipNull() {
+ currentIndex++;
+ }
+
+ @Override
+ public void setPosition(int index) {
+ currentIndex = index;
+ }
+
+ @Override
+ public void close() throws Exception {
+ vector.close();
+ }
+
+ @Override
+ public boolean resetValueVector(T vector) {
+ this.vector = vector;
+ this.currentIndex = 0;
+ return true;
+ }
Review Comment:
I think I agree with you that producers should not own / close the vectors. In which case, this becomes a non-issue! Are you happy for me to update on those lines? The producers will still own a reference, but we could get rid of all the close() methods and remove AutoClosable from the base interface. Would you like me to do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1976671881
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/CompositeAvroProducer.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.avro.io.Encoder;
+
+/** Composite producer which holds all producers. It manages the produce and cleanup process. */
+public class CompositeAvroProducer implements AutoCloseable {
+
+ private final List<Producer<? extends FieldVector>> producers;
+
+ public CompositeAvroProducer(List<Producer<? extends FieldVector>> producers) {
+ this.producers = producers;
+ }
+
+ public List<Producer<?>> getProducers() {
+ return producers;
+ }
+
+ /** Produce encoder data. */
+ public void produce(Encoder encoder) throws IOException {
+ for (Producer<? extends FieldVector> producer : producers) {
+ producer.produce(encoder);
+ }
+ }
+
+ /** Reset vector of consumers with the given {@link VectorSchemaRoot}. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public void resetProducerVectors(VectorSchemaRoot root) {
+ // This method assumes that the VSR matches the constructed set of producers
+ int index = 0;
+ for (Producer producer : producers) {
+ if (producer.resetValueVector(root.getFieldVectors().get(index))) {
+ index++;
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ // clean up
+ try {
+ AutoCloseables.close(producers);
+ } catch (Exception e) {
+ throw new RuntimeException("Error occurs in close.", e);
Review Comment:
Per above, close and AutoCloable are gone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010701197
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/BaseAvroProducer.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+
+/**
+ * Base class for avro producers.
+ *
+ * @param <T> vector type.
+ */
+public abstract class BaseAvroProducer<T extends FieldVector> implements Producer<T> {
+
+ protected T vector;
+ protected int currentIndex;
+
+ /**
+ * Constructs a base avro consumer.
+ *
+ * @param vector the vector to consume.
+ */
+ protected BaseAvroProducer(T vector) {
+ this.vector = vector;
+ }
+
+ @Override
+ public void skipNull() {
+ currentIndex++;
+ }
+
+ @Override
+ public void setPosition(int index) {
+ currentIndex = index;
Review Comment:
I have added check, index < 0 or index > value count will throw an illegal argument exception.
I've allowed index == value count, because that is the final state after production is complete. Since the producer can get into that state naturally, and it has a valid meaning (production complete), it seemed wrong to disallow it. Thoughts welcome on that point!
Worth noting, there are several ways value count can not be set on the vector, either because client code doesn't set it, or a writer doesn't (e.g. UnionMapWriter does not update the entries data vector count) in which case the cause of the error may not be obvious (at least it wasn't for me)! Probably still the check is the right thing to do, but something to be aware of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2753580739
I think the PR check hit [this issue](https://github.com/apache/arrow-java/pull/673) - could you add the milestone to the issue manually for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1980767717
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroStructProducer.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer which produces nested record type values to avro encoder. Read the data from {@link
+ * org.apache.arrow.vector.complex.StructVector}.
+ */
+public class AvroStructProducer extends BaseAvroProducer<StructVector> {
+
+ private final Producer<? extends FieldVector>[] delegates;
+
+ /** Instantiate a AvroStructProducer. */
+ public AvroStructProducer(StructVector vector, Producer<? extends FieldVector>[] delegates) {
+ super(vector);
+ this.delegates = delegates;
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+
+ for (Producer<?> delegate : delegates) {
+ delegate.produce(encoder);
+ }
+
+ currentIndex++;
+ }
+
+ @Override
+ public void skipNull() {
+ for (Producer<?> delegate : delegates) {
+ delegate.skipNull();
+ }
+ super.skipNull();
+ }
+
+ @Override
+ public void setPosition(int index) {
+ for (Producer<?> delegate : delegates) {
+ delegate.setPosition(index);
+ }
+ super.setPosition(index);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public boolean resetValueVector(StructVector vector) {
+ for (int i = 0; i < delegates.length; i++) {
+ Producer<FieldVector> delegate = (Producer<FieldVector>) delegates[i];
+ delegate.resetValueVector(vector.getChildrenFromFields().get(i));
Review Comment:
Should the return value here be accounted for?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDenseUnionProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.DenseUnionVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <ul>
+ * <li>ArrowType.Null --> NULL
Review Comment:
not to nitpick the formatting but a table might be better? (also does Javadoc allow unclosed tags like this?)
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/BaseTimestampTzProducer.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.vector.TimeStampVector;
+import org.apache.avro.io.Encoder;
+
+abstract class BaseTimestampTzProducer<T extends TimeStampVector>
+ extends BaseAvroProducer<TimeStampVector> {
+
+ // Convert TZ values to UTC to encode Avro timestamp types
Review Comment:
Arrow stores zoned timestamps in UTC already. The time zone associated with the type is purely used for display. So there should be no need for conversion.
https://github.com/apache/arrow/blob/601fde9033b5def7e667ffec70c0acacd1b9ad50/format/Schema.fbs#L284-L302
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroUint4Producer.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces long values from a {@link UInt4Vector}, writes data to an avro encoder.
+ */
+public class AvroUint4Producer extends BaseAvroProducer<UInt4Vector> {
+
+ /** Instantiate an AvroUint4Producer. */
+ public AvroUint4Producer(UInt4Vector vector) {
+ super(vector);
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ int unsigned = vector.getDataBuffer().getInt(currentIndex * (long) UInt4Vector.TYPE_WIDTH);
+ long signed = unsigned & 0xffffffffL;
Review Comment:
Is this https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html#toUnsignedLong-int- ?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Interface that is used to produce values to avro encoder.
+ *
+ * @param <T> The vector within producer or its delegate, used for partially produce purpose.
+ */
+public interface Producer<T extends FieldVector> {
+
+ /**
+ * Produce a specific type value from the vector and write it to avro encoder.
+ *
+ * @param encoder avro encoder to write data
+ * @throws IOException on error
+ */
+ void produce(Encoder encoder) throws IOException;
+
+ /** Skip null value in the vector by setting reader position + 1. */
+ void skipNull();
+
+ /** Set the position to read value from vector. */
+ void setPosition(int index);
+
+ /**
+ * Reset the vector within producer.
+ *
+ * @return true if reset is successful, false if reset is not needed.
+ */
+ boolean resetValueVector(T vector);
Review Comment:
Hmm, do any of the implementations ever return `false`?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroTimeMilliProducer.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.vector.TimeMilliVector;
+
+/**
+ * Producer that produces time (milliseconds) values from a {@link TimeMilliVector}, writes data to
+ * an Avro encoder.
+ */
+public class AvroTimeMilliProducer extends AvroIntProducer {
+
+ // Time in milliseconds stored as integer, matches Avro time-millis type
Review Comment:
Hmm, Arrow timestamp type is always 8 bytes, so won't this suffer a range issue?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+import java.io.IOException;
+
+/**
+ * Interface that is used to produce values to avro encoder.
+ *
+ * @param <T> The vector within producer or its delegate, used for partially produce purpose.
+ */
+public interface Producer<T extends FieldVector> extends AutoCloseable {
+
+ /**
+ * Produce a specific type value from the vector and write it to avro encoder.
+ *
+ * @param encoder avro encoder to write data
+ * @throws IOException on error
+ */
+ void produce(Encoder encoder) throws IOException;
+
+ /** Skip null value in the vector by setting reader position + 1. */
+ void skipNull();
+
+ /** Set the position to read value from vector. */
+ void setPosition(int index);
Review Comment:
Ok, let's leave it then. (We really need to fix this more generally...)
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/AvroUint2Producer.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import java.io.IOException;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.avro.io.Encoder;
+
+/** Producer that produces int values from a {@link UInt2Vector}, writes data to an avro encoder. */
+public class AvroUint2Producer extends BaseAvroProducer<UInt2Vector> {
+
+ /** Instantiate an AvroUint2Producer. */
+ public AvroUint2Producer(UInt2Vector vector) {
+ super(vector);
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ short unsigned = vector.getDataBuffer().getShort(currentIndex * (long) UInt2Vector.TYPE_WIDTH);
+ int signed = unsigned & 0xffff;
Review Comment:
Same here, is Short#toUnsignedInt applicable?
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroBigIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDenseUnionProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeBinaryProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedSizeListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroFloat8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroListProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroSmallIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroTinyIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUint1Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint2Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint4Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUint8Producer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateDayProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimal256Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMilliTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampNanoTzProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampSecTzProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DateMilliVector;
+import org.apache.arrow.vector.Decimal256Vector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float2Vector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.arrow.vector.TimeSecVector;
+import org.apache.arrow.vector.TimeStampMicroTZVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliTZVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.TimeStampNanoTZVector;
+import org.apache.arrow.vector.TimeStampNanoVector;
+import org.apache.arrow.vector.TimeStampSecTZVector;
+import org.apache.arrow.vector.TimeStampSecVector;
+import org.apache.arrow.vector.TinyIntVector;
+import org.apache.arrow.vector.UInt1Vector;
+import org.apache.arrow.vector.UInt2Vector;
+import org.apache.arrow.vector.UInt4Vector;
+import org.apache.arrow.vector.UInt8Vector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.DenseUnionVector;
+import org.apache.arrow.vector.complex.FixedSizeListVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+
+public class ArrowToAvroUtils {
+
+ public static final String GENERIC_RECORD_TYPE_NAME = "GenericRecord";
+
+ /**
+ * Create an Avro record schema for a given list of Arrow fields.
+ *
+ * <p>This method currently performs following type mapping for Avro data types to corresponding
+ * Arrow data types.
+ *
+ * <ul>
+ * <li>ArrowType.Null --> NULL
+ * <li>ArrowType.Bool --> BOOLEAN
+ * <li>ArrowType.Int(64 bit, unsigned 32 bit) --> LONG
+ * <li>ArrowType.Int(signed 32 bit, < 32 bit) --> INT
+ * <li>ArrowType.FloatingPoint(double) --> DOUBLE
+ * <li>ArrowType.FloatingPoint(single, half) --> FLOAT
+ * <li>ArrowType.Utf8 --> STRING
+ * <li>ArrowType.LargeUtf8 --> STRING
+ * <li>ArrowType.Binary --> BYTES
+ * <li>ArrowType.LargeBinary --> BYTES
+ * <li>ArrowType.FixedSizeBinary --> FIXED
+ * <li>ArrowType.Decimal --> decimal (FIXED)
+ * <li>ArrowType.Date --> date (INT)
+ * <li>ArrowType.Time (MILLI) --> time-millis (INT)
+ * <li>ArrowType.Time (SEC | MICRO | NANO) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ != NULL) --> time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ != NULL) --> time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ != NULL) --> time-millis (LONG)
+ * <li>ArrowType.Timestamp (NANOSECONDS, TZ == NULL) --> local-time-nanos (LONG)
+ * <li>ArrowType.Timestamp (MICROSECONDS, TZ == NULL) --> local-time-micros (LONG)
+ * <li>ArrowType.Timestamp (MILLISECONDS | SECONDS, TZ == NULL) --> local-time-millis (LONG)
+ * <li>ArrowType.Duration --> duration (FIXED)
+ * <li>ArrowType.Interval --> duration (FIXED)
+ * <li>ArrowType.Struct --> record
+ * <li>ArrowType.List --> array
+ * <li>ArrowType.LargeList --> array
+ * <li>ArrowType.FixedSizeList --> array
+ * <li>ArrowType.Map --> map
+ * <li>ArrowType.Union --> union
+ * </ul>
+ *
+ * <p>Nullable fields are represented as a union of [null | base-type]. Special treatment is given
+ * to nullability of unions - a union is considered nullable if the union field is nullable or any
Review Comment:
FWIW, I think nullable unions aren't semantically valid - unions have no validity bitmap in Arrow. The only way to represent this is with a nullable child field
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroTimeNanoProducer.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that converts nanoseconds from a {@link TimeNanoVector} and produces time (microseconds)
+ * values, writes data to an Avro encoder.
+ */
+public class AvroTimeNanoProducer extends BaseAvroProducer<TimeNanoVector> {
+
+ // Convert nanoseconds to microseconds for Avro time-micros (LONG) type
+ // Range is 1000 times less than for microseconds, so the type will fit (with loss of precision)
+
+ private static final long NANOS_PER_MICRO = 1000;
+
+ public AvroTimeNanoProducer(TimeNanoVector vector) {
+ super(vector);
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ long nanos = vector.getDataBuffer().getLong(currentIndex * (long) TimeNanoVector.TYPE_WIDTH);
+ long micros = nanos / NANOS_PER_MICRO;
Review Comment:
Maybe we should throw on truncation to be safe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2771599238
Hi @lidavidm - Thanks so much for merging, I'm really happy to get this first piece in :-) Have started work on the second bit, once the main pieces are ready I will raise a new PR.
I'm snowed under too so may have to put it down and come back a week later etc., very happy to work in async mode!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2755511607
> Please ignore the check 😬
Ok great - in that case I think the CI is done :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #638:
URL: https://github.com/apache/arrow-java/pull/638#issuecomment-2753625036
Please ignore the check :grimacing:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010701197
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/BaseAvroProducer.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+
+/**
+ * Base class for avro producers.
+ *
+ * @param <T> vector type.
+ */
+public abstract class BaseAvroProducer<T extends FieldVector> implements Producer<T> {
+
+ protected T vector;
+ protected int currentIndex;
+
+ /**
+ * Constructs a base avro consumer.
+ *
+ * @param vector the vector to consume.
+ */
+ protected BaseAvroProducer(T vector) {
+ this.vector = vector;
+ }
+
+ @Override
+ public void skipNull() {
+ currentIndex++;
+ }
+
+ @Override
+ public void setPosition(int index) {
+ currentIndex = index;
Review Comment:
I have added check, index < 0 or index > value count will throw an illegal argument exception.
I've allowed index == value count, because that is the final state after production is complete. Since the producer can get into that state naturally, and it has a valid meaning (production complete), it seemed wrong to disallow it. Thoughts welcome on that point!
Worth noting, there are several ways value count can not be set on the vector, either because client code doesn't set it, or because some code paths that probably should set it don't (e.g. FieldWriter for Map entries). Users may encounter problems because of that (I did)! Probably still the check is the right thing to do, but something to be aware of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010629007
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/ArrowToAvroUtils.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.arrow.adapter.avro.producers.AvroArraysProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBooleanProducer;
+import org.apache.arrow.adapter.avro.producers.AvroBytesProducer;
+import org.apache.arrow.adapter.avro.producers.AvroDoubleProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFixedProducer;
+import org.apache.arrow.adapter.avro.producers.AvroFloatProducer;
+import org.apache.arrow.adapter.avro.producers.AvroIntProducer;
+import org.apache.arrow.adapter.avro.producers.AvroLongProducer;
+import org.apache.arrow.adapter.avro.producers.AvroMapProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullProducer;
+import org.apache.arrow.adapter.avro.producers.AvroNullableProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStringProducer;
+import org.apache.arrow.adapter.avro.producers.AvroStructProducer;
+import org.apache.arrow.adapter.avro.producers.AvroUnionsProducer;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.adapter.avro.producers.CompositeAvroProducer;
+import org.apache.arrow.adapter.avro.producers.Producer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDateProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroDecimalProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimeMillisProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMicroProducer;
+import org.apache.arrow.adapter.avro.producers.logical.AvroTimestampMillisProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.DateDayVector;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.FixedSizeBinaryVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.NullVector;
+import org.apache.arrow.vector.TimeMicroVector;
+import org.apache.arrow.vector.TimeMilliVector;
+import org.apache.arrow.vector.TimeStampMicroVector;
+import org.apache.arrow.vector.TimeStampMilliVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.complex.MapVector;
+import org.apache.arrow.vector.complex.StructVector;
+import org.apache.arrow.vector.complex.UnionVector;
+import org.apache.arrow.vector.types.Types;
+
+public class ArrowToAvroUtils {
+
+ /**
+ * Create a composite Avro producer for a set of field vectors (typically the root set of a VSR).
+ *
+ * @param vectors The vectors that will be used to produce Avro data
+ * @return The resulting composite Avro producer
+ */
+ public static CompositeAvroProducer createCompositeProducer(List<FieldVector> vectors) {
+
+ List<Producer<? extends FieldVector>> producers = new ArrayList<>(vectors.size());
+
+ for (FieldVector vector : vectors) {
+ BaseAvroProducer<? extends FieldVector> producer = createProducer(vector);
+ producers.add(producer);
+ }
+
+ return new CompositeAvroProducer(producers);
+ }
+
+ private static BaseAvroProducer<?> createProducer(FieldVector vector) {
+ boolean nullable = vector.getField().isNullable();
+ return createProducer(vector, nullable);
+ }
+
+ private static BaseAvroProducer<?> createProducer(FieldVector vector, boolean nullable) {
+
+ Preconditions.checkNotNull(vector, "Arrow vector object can't be null");
+
+ final Types.MinorType minorType = vector.getMinorType();
+
+ // Avro understands nullable types as a union of type | null
+ // Most nullable fields in a VSR will not be unions, so provide a special wrapper
+ if (nullable && minorType != Types.MinorType.UNION) {
+ final BaseAvroProducer<?> innerProducer = createProducer(vector, false);
+ return new AvroNullableProducer<>(innerProducer);
+ }
+
+ switch (minorType) {
+
+ // Primitive types with direct mapping to Avro
+
+ case NULL:
+ return new AvroNullProducer((NullVector) vector);
+ case BIT:
+ return new AvroBooleanProducer((BitVector) vector);
+ case INT:
+ return new AvroIntProducer((IntVector) vector);
+ case BIGINT:
+ return new AvroLongProducer((BigIntVector) vector);
+ case FLOAT4:
+ return new AvroFloatProducer((Float4Vector) vector);
+ case FLOAT8:
+ return new AvroDoubleProducer((Float8Vector) vector);
+ case VARBINARY:
+ return new AvroBytesProducer((VarBinaryVector) vector);
+ case FIXEDSIZEBINARY:
+ return new AvroFixedProducer((FixedSizeBinaryVector) vector);
+ case VARCHAR:
+ return new AvroStringProducer((VarCharVector) vector);
+
+ // Logical types
+
+ case DECIMAL:
+ return new AvroDecimalProducer.FixedDecimalProducer(
+ (DecimalVector) vector, DecimalVector.TYPE_WIDTH);
+ case DATEDAY:
+ return new AvroDateProducer((DateDayVector) vector);
+ case TIMEMILLI:
+ return new AvroTimeMillisProducer((TimeMilliVector) vector);
+ case TIMEMICRO:
+ return new AvroTimeMicroProducer((TimeMicroVector) vector);
+ case TIMESTAMPMILLI:
+ return new AvroTimestampMillisProducer((TimeStampMilliVector) vector);
+ case TIMESTAMPMICRO:
+ return new AvroTimestampMicroProducer((TimeStampMicroVector) vector);
+
+ // Complex types
+
+ case STRUCT:
+ StructVector structVector = (StructVector) vector;
+ List<FieldVector> childVectors = structVector.getChildrenFromFields();
+ Producer<?>[] childProducers = new Producer<?>[childVectors.size()];
+ for (int i = 0; i < childVectors.size(); i++) {
+ FieldVector childVector = childVectors.get(i);
+ childProducers[i] = createProducer(childVector, childVector.getField().isNullable());
+ }
+ return new AvroStructProducer(structVector, childProducers);
+
+ case LIST:
+ ListVector listVector = (ListVector) vector;
+ FieldVector itemVector = listVector.getDataVector();
+ Producer<?> itemProducer = createProducer(itemVector, itemVector.getField().isNullable());
+ return new AvroArraysProducer(listVector, itemProducer);
+
+ case MAP:
+ MapVector mapVector = (MapVector) vector;
+ StructVector entryVector = (StructVector) mapVector.getDataVector();
+ VarCharVector keyVector = (VarCharVector) entryVector.getChildrenFromFields().get(0);
Review Comment:
I added a check for this, now it will throw illegal argument if the key type is not string (which is required by Avro)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010637729
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroTimeNanoProducer.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.vector.TimeNanoVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that converts nanoseconds from a {@link TimeNanoVector} and produces time (microseconds)
+ * values, writes data to an Avro encoder.
+ */
+public class AvroTimeNanoProducer extends BaseAvroProducer<TimeNanoVector> {
+
+ // Convert nanoseconds to microseconds for Avro time-micros (LONG) type
+ // Range is 1000 times less than for microseconds, so the type will fit (with loss of precision)
+
+ private static final long NANOS_PER_MICRO = 1000;
+
+ public AvroTimeNanoProducer(TimeNanoVector vector) {
+ super(vector);
+ }
+
+ @Override
+ public void produce(Encoder encoder) throws IOException {
+ long nanos = vector.getDataBuffer().getLong(currentIndex * (long) TimeNanoVector.TYPE_WIDTH);
+ long micros = nanos / NANOS_PER_MICRO;
Review Comment:
I don't think this will truncate since we are going nanos -> micros so there is more range available. There is loss of precision - I think that is acceptable, since Avro does not have a time-nanos type (at least not yet)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1974724340
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/AvroDecimalProducer.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.DecimalVector;
+import org.apache.avro.io.Encoder;
+
+/**
+ * Producer that produces decimal values from a {@link DecimalVector}, writes data to an Avro
+ * encoder.
+ */
+public abstract class AvroDecimalProducer extends BaseAvroProducer<DecimalVector> {
+
+ /** Instantiate an AvroDecimalProducer. */
+ protected AvroDecimalProducer(DecimalVector vector) {
+ super(vector);
+ }
+
+ /** Producer for decimal logical type with original bytes type. */
+ public static class BytesDecimalProducer extends AvroDecimalProducer {
+
+ private final byte[] reuseBytes;
+
+ /** Instantiate a BytesDecimalConsumer. */
+ public BytesDecimalProducer(DecimalVector vector) {
+ super(vector);
+ Preconditions.checkArgument(
+ vector.getTypeWidth() <= 16, "Decimal bytes length should <= 16.");
+ reuseBytes = new byte[vector.getTypeWidth()];
Review Comment:
Decimal is fixed width byte (either 16 or 32)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r1971815360
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/Producer.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers;
+
+import org.apache.arrow.vector.FieldVector;
+import org.apache.avro.io.Encoder;
+
+import java.io.IOException;
+
+/**
+ * Interface that is used to produce values to avro encoder.
+ *
+ * @param <T> The vector within producer or its delegate, used for partially produce purpose.
+ */
+public interface Producer<T extends FieldVector> extends AutoCloseable {
+
+ /**
+ * Produce a specific type value from the vector and write it to avro encoder.
+ *
+ * @param encoder avro encoder to write data
+ * @throws IOException on error
+ */
+ void produce(Encoder encoder) throws IOException;
+
+ /** Skip null value in the vector by setting reader position + 1. */
+ void skipNull();
+
+ /** Set the position to read value from vector. */
+ void setPosition(int index);
Review Comment:
Ok, that makes sense. We should really update the consumers to match. If you like, I could make both int and long available as overloads, and keep a long internally. That way we can update the consumer interface to match, without breaking the API. Would you like me to do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on code in PR #638:
URL: https://github.com/apache/arrow-java/pull/638#discussion_r2010643616
##########
adapter/avro/src/main/java/org/apache/arrow/adapter/avro/producers/logical/BaseTimestampTzProducer.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.adapter.avro.producers.logical;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import org.apache.arrow.adapter.avro.producers.BaseAvroProducer;
+import org.apache.arrow.vector.TimeStampVector;
+import org.apache.avro.io.Encoder;
+
+abstract class BaseTimestampTzProducer<T extends TimeStampVector>
+ extends BaseAvroProducer<TimeStampVector> {
+
+ // Convert TZ values to UTC to encode Avro timestamp types
Review Comment:
This is great! I took out all the timezone conversion logic, the TZ aware producers are much simpler now :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-615: Produce Avro core data types out of Arrow VSR [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #638:
URL: https://github.com/apache/arrow-java/pull/638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org