You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/29 19:58:58 UTC
[GitHub] [arrow-cookbook] mystic-lama opened a new pull request, #235: [Java] Added section for converting Avro To Arrow
mystic-lama opened a new pull request, #235:
URL: https://github.com/apache/arrow-cookbook/pull/235
Added example to convert Avro to Arrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937148656
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.IOException;
+
+ try {
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ try (BufferAllocator allocator = new RootAllocator()) {
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+ try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
+ while(avroToArrowVectorIterator.hasNext()) {
+ try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
+ System.out.print(root.contentToTSVString());
+ }
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
Review Comment:
It doesn't appear to have changed, additionally the code and output are still misaligned by one space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] mystic-lama commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
mystic-lama commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937163798
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.IOException;
+
+ try {
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ try (BufferAllocator allocator = new RootAllocator()) {
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+ try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
+ while(avroToArrowVectorIterator.hasNext()) {
+ try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
+ System.out.print(root.contentToTSVString());
+ }
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
Review Comment:
pushed the change, one issue i noted it the html rendered output has extra space between results, not sure where it is coming from. Here is how it looks in html
name favorite_number favorite_color
Alyssa 256 null
Ben 7 red
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lwhite1 commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lwhite1 commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1204408572
> The Avro adapter isn't part of datasets.
ORC isn't part of datasets either (in Java, currently)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] mystic-lama commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
mystic-lama commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937155057
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.IOException;
+
+ try {
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ try (BufferAllocator allocator = new RootAllocator()) {
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+ try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
+ while(avroToArrowVectorIterator.hasNext()) {
+ try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
+ System.out.print(root.contentToTSVString());
+ }
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
Review Comment:
I am just following format similar to other files, hence didn't remove extra space char. Should I align? Should we do it for other files too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] davisusanibar commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
davisusanibar commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1204318493
> The Avro adapter isn't part of datasets.
It means we could continue as it is now (maybe change title from 'Avro to Arrow' to 'Avro' only?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] davisusanibar commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
davisusanibar commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1204226497
I have a question about how to organize our recipes for dataset such as: parquet / avro / orc
With this PR this will be the result:
<img width="220" alt="image" src="https://user-images.githubusercontent.com/4554485/182665013-a6304f1b-feda-4001-b83a-873cbaef6964.png">
What about if we could organize dataset cookbooks on this way?
<img width="412" alt="image" src="https://user-images.githubusercontent.com/4554485/182665184-21ab62a8-a3d2-4c39-a57c-a685f4025ec4.png">
Please if you could help me on this validation @lidavidm / @lwhite1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1204398501
Yeah, if we had other formats, they could be their own section, but it's just Avro and Arrow IPC itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] mystic-lama commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
mystic-lama commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937000678
##########
java/source/avro.rst:
##########
@@ -0,0 +1,67 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+
+ BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config);
+
+ while(avroToArrowVectorIterator.hasNext()) {
+ VectorSchemaRoot root = avroToArrowVectorIterator.next();
+ System.out.print(root.contentToTSVString());
+ }
Review Comment:
@davisusanibar This issue seems to be common theme from my side. How do I ensure I don't make the same mistake again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] mystic-lama commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
mystic-lama commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937116326
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
Review Comment:
Arrow data file can store schema along with data. This is in reference to that. This example shall fail in that case, hence mentioned explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1204495529
@lwhite1 all examples are given for JShell and are runnable in that environment (`make javatest` will actually run the code and compare the results). Maybe we need to make that more explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] mystic-lama commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
mystic-lama commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1199946177
@lidavidm Can use your help with this PR. Still at beginning phases of learning Arrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] davisusanibar commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
davisusanibar commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r936985039
##########
java/source/avro.rst:
##########
@@ -0,0 +1,67 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+
+ BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config);
+
+ while(avroToArrowVectorIterator.hasNext()) {
+ VectorSchemaRoot root = avroToArrowVectorIterator.next();
+ System.out.print(root.contentToTSVString());
+ }
Review Comment:
```suggestion
import org.apache.arrow.AvroToArrow;
import org.apache.arrow.AvroToArrowConfig;
import org.apache.arrow.AvroToArrowConfigBuilder;
import org.apache.arrow.AvroToArrowVectorIterator;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.avro.Schema;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.DecoderFactory;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
try {
BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
try (BufferAllocator allocator = new RootAllocator()) {
AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
while(avroToArrowVectorIterator.hasNext()) {
try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
System.out.print(root.contentToTSVString());
}
}
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937155768
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.IOException;
+
+ try {
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ try (BufferAllocator allocator = new RootAllocator()) {
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+ try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
+ while(avroToArrowVectorIterator.hasNext()) {
+ try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
+ System.out.print(root.contentToTSVString());
+ }
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
Review Comment:
Oops. It should generally be aligned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937067684
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.IOException;
+
+ try {
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ try (BufferAllocator allocator = new RootAllocator()) {
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+ try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
+ while(avroToArrowVectorIterator.hasNext()) {
+ try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
+ System.out.print(root.contentToTSVString());
+ }
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
Review Comment:
Consolidate the catches?
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
Review Comment:
Ah, separate files.
```suggestion
The example assumes that the Avro schema is stored separately from the Avro data itself.
```
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
Review Comment:
I honestly do not know what this means? The code sample is reading a schema it appears
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm merged pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm merged PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lwhite1 commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lwhite1 commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1204423991
With David's suggestions, this looks good to me.
FWIW, I find the approach of providing sufficient code for the reader to create a runnable example, without actually providing a runnable (or compilable) example a little odd, but it seems to be the standard approach for the Cookbook. And it is a little shorter :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1201106862
@davisusanibar would you like to review as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] lidavidm commented on pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#issuecomment-1204289544
The Avro adapter isn't part of datasets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] mystic-lama commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
mystic-lama commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937144350
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.IOException;
+
+ try {
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ try (BufferAllocator allocator = new RootAllocator()) {
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+ try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
+ while(avroToArrowVectorIterator.hasNext()) {
+ try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
+ System.out.print(root.contentToTSVString());
+ }
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-cookbook] mystic-lama commented on a diff in pull request #235: [Java] Added section for converting Avro To Arrow
Posted by GitBox <gi...@apache.org>.
mystic-lama commented on code in PR #235:
URL: https://github.com/apache/arrow-cookbook/pull/235#discussion_r937156687
##########
java/source/avro.rst:
##########
@@ -0,0 +1,75 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _arrow-avro:
+
+===============================
+Avro to Arrow
+===============================
+
+Avro encoded data can be converted into Arrow format.
+
+.. contents::
+
+Avro to Arrow
+=============
+
+The example assumes that the avro data doesn't contain schema in the file
+to be converted into Arrow
+
+.. testcode::
+
+ import org.apache.arrow.AvroToArrow;
+ import org.apache.arrow.AvroToArrowConfig;
+ import org.apache.arrow.AvroToArrowConfigBuilder;
+ import org.apache.arrow.AvroToArrowVectorIterator;
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.avro.Schema;
+ import org.apache.avro.io.BinaryDecoder;
+ import org.apache.avro.io.DecoderFactory;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.IOException;
+
+ try {
+ BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("./thirdpartydeps/avro/users.avro"), null);
+ Schema schema = new Schema.Parser().parse(new File("./thirdpartydeps/avro/user.avsc"));
+ try (BufferAllocator allocator = new RootAllocator()) {
+ AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
+ try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
+ while(avroToArrowVectorIterator.hasNext()) {
+ try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
+ System.out.print(root.contentToTSVString());
+ }
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
Review Comment:
Let me go ahead and fix the alignment, if it looks good we shall take care in other files when we touch them. Give me 5 mins
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org