You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/30 20:29:00 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
wjones127 opened a new pull request, #14008:
URL: https://github.com/apache/arrow/pull/14008
Adds a little more explanation of the difference between tables and record batches, as well as a diagram representation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960314403
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
+
+ .. image:: ../cpp/tables-versus-record-batches.svg
Review Comment:
For the record, is there any source file to this SVG (I don't know how it was generated)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
drin commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r958951760
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
+in the Arrow format itself, so they aren't directly portable.
+
+However, a table can be converted to and built from a sequence of record
+batches easily without needing to copy the underlying array buffers.
Review Comment:
If the columns have different chunking, wouldn't it be necessary to normalize when converting to a RecordBatch?
That being said, a Table is basically a structural view of a RecordBatch: each RecordBatch Array can be wrapped in a ChunkedArray trivially because the buffers are separate and a ChunkedArray is *mostly* an ArrayVector (though, I assume not entirely).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960747507
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,18 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Record batches can be sent between implementations, such as via
+:ref:`IPC <format-ipc>` or
+via the :doc:`C Data Interface <../format/CDataInterface>`. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation (and
+its bindings), not in the Arrow format itself, so they aren't directly portable.
Review Comment:
```suggestion
chunked arrays, on the other hand, are concepts in the C++ implementation,
not in the Arrow format itself, so they aren't directly portable.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r959287295
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
+in the Arrow format itself, so they aren't directly portable.
+
+However, a table can be converted to and built from a sequence of record
+batches easily without needing to copy the underlying array buffers.
Review Comment:
We don't have any function to re-chunk more reasonably, no. I think heterogeneous chunkings should be uncommon in general, anyway, since all data sources should yield homogenous chunkings (i.e. a stream of RecordBatch).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r959862298
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,18 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Record batches can be sent between implementations, such as via
+:ref:`IPC <format-ipc>` or
+via the :doc:`C Data Interface <../format/CDataInterface>`. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation (and
+its bindings), not in the Arrow format itself, so they aren't directly portable.
Review Comment:
This is within the C++ user guide, so I'm not too worried about Python or R users reading this and not being sure if this applies to them. Presumably they would only be on this page if they already knew!
But we should create a diagram on the Docs homepage that shows the relationship between implementations. I'll leave that as a separate ticket though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14008:
URL: https://github.com/apache/arrow/pull/14008#issuecomment-1232133972
https://issues.apache.org/jira/browse/ARROW-13454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r958912212
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
Review Comment:
Also, the glossary talks about this as well, dunno if that's useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r958925250
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
Review Comment:
Yeah maybe it should be C++ and it's bindings (Python is one of those).
Another diagram I'd like to have is which implementations are base ones (e.g. C++, Rust), and which are bindings (and to whom). I know where C++/Python/R/Ruby, C#, and Rust sit. I'm still unclear where Go and Java are though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960315265
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
Review Comment:
Let's be less specific perhaps, and say "for example C++ and its bindings"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960741137
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
+
+ .. image:: ../cpp/tables-versus-record-batches.svg
Review Comment:
The SVG is the source file; I wrote it by hand. It's just `rect` and `text` for the most part and I added the Apache license header as a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14008:
URL: https://github.com/apache/arrow/pull/14008#issuecomment-1232133996
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960744140
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
+
+ .. image:: ../cpp/tables-versus-record-batches.svg
Review Comment:
I didn't know people wrote SVG files by hand :-o
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960744128
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
Review Comment:
```suggestion
certain language implementations of Arrow (for example C++ and
its bindings, and Go).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r958956177
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
+in the Arrow format itself, so they aren't directly portable.
+
+However, a table can be converted to and built from a sequence of record
+batches easily without needing to copy the underlying array buffers.
Review Comment:
> If the columns have different chunking, wouldn't it be necessary to normalize when converting to a RecordBatch?
`TableBatchReader` will create a RB boundary any place there is a chunk boundary. So if you have lots of different chunking you will get lots of uneven batches. I imagine there's some function to re-chunk more reasonably if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou merged pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou merged PR #14008:
URL: https://github.com/apache/arrow/pull/14008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r975065317
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
Review Comment:
Definitely, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r974619281
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
Review Comment:
I wrote "(for example C++ and its bindings, and Go).", since I felt that Go was a useful detail. Is that acceptable to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
drin commented on PR #14008:
URL: https://github.com/apache/arrow/pull/14008#issuecomment-1232198649
Not sure if it's a significant thing, but a Table can easily modify schema metadata whereas a recordbatch cannot (I'm also not sure if there's a reason RecordBatch doesn't have `ReplaceSchemaMetadata`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960744128
##########
docs/source/format/Glossary.rst:
##########
@@ -197,6 +197,10 @@ Glossary
Not part of the columnar format; this term is specific to
certain language implementations of Arrow (primarily C++ and
- its bindings).
+ its bindings and Go).
Review Comment:
```suggestion
certain language implementations of Arrow (for example C++ and
its bindings, and Go).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r959860181
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
+in the Arrow format itself, so they aren't directly portable.
+
+However, a table can be converted to and built from a sequence of record
+batches easily without needing to copy the underlying array buffers.
Review Comment:
> Does each RecordBatch just store pointers to start and end, and the breaking each boundary ensures contiguity?
Arrays generally have a `shared_ptr<Buffer>`, an offset, and a length. You can `Slice()` any array, and it just copies the shared pointer to the buffer, and adjusts the offset and length accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r960313802
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,18 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Record batches can be sent between implementations, such as via
+:ref:`IPC <format-ipc>` or
+via the :doc:`C Data Interface <../format/CDataInterface>`. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation (and
+its bindings), not in the Arrow format itself, so they aren't directly portable.
Review Comment:
Let's perhaps remove the mention of bindings then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r958926118
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
Review Comment:
Go and Java are standalone implementations like C++/C♯/Rust/Julia (Java was one of the original implementations)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
westonpace commented on PR #14008:
URL: https://github.com/apache/arrow/pull/14008#issuecomment-1232318340
> Not sure if it's a significant thing, but a Table can easily modify schema metadata whereas a recordbatch cannot (I'm also not sure if there's a reason RecordBatch doesn't have ReplaceSchemaMetadata).
I think that is just a gap in utility APIs. `ReplaceSchemaMetadata` isn't really modifying the table. It's creating a brand new table with the new metadata. You should be able to do the same thing with record batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r958911296
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
Review Comment:
Do we want to assume knowledge of the C Data Interface?
We could make this more generic: "they can be sent between implementations, such as via IPC (link), or via the C Data Interface (link)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ksuarez1423 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
ksuarez1423 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r959841323
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,18 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Record batches can be sent between implementations, such as via
+:ref:`IPC <format-ipc>` or
+via the :doc:`C Data Interface <../format/CDataInterface>`. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation (and
+its bindings), not in the Arrow format itself, so they aren't directly portable.
Review Comment:
Can we link up to the list of bindings somewhere, or should we expect readers to know if they're using a binded or fully-implemented implementation?
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
+in the Arrow format itself, so they aren't directly portable.
+
+However, a table can be converted to and built from a sequence of record
+batches easily without needing to copy the underlying array buffers.
Review Comment:
What exactly happens, if you break at every boundary? Does each RecordBatch just store pointers to start and end, and the breaking each boundary ensures contiguity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14008:
URL: https://github.com/apache/arrow/pull/14008#issuecomment-1252253867
Benchmark runs are scheduled for baseline = cd67e5195aee78748e2b646a15a4e9f8b791a776 and contender = ab71673ce0955798645ae9178018f562a82ed7f2. ab71673ce0955798645ae9178018f562a82ed7f2 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/050fbd16a53249e4b57bec7201324a33...ec512d413f4a46d887456c73e1ee9e4d/)
[Failed :arrow_down:0.2% :arrow_up:0.03%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/e007828c729a4ae08b2e5c3f458e4a20...679ffc320380479482836b23f082054c/)
[Failed :arrow_down:0.28% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/4ca0ea494c944307a858d1156ebb090d...4ec383f5bc4a4802b48c3a1103f17128/)
[Finished :arrow_down:0.14% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/23304097403e44b8a6ff6286cc00b77c...ba8f54fae9624c82ac12c5e9c3fc4f6e/)
Buildkite builds:
[Finished] [`ab71673c` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1522)
[Failed] [`ab71673c` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1539)
[Failed] [`ab71673c` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1522)
[Finished] [`ab71673c` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1536)
[Finished] [`cd67e519` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1521)
[Failed] [`cd67e519` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1538)
[Failed] [`cd67e519` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1521)
[Finished] [`cd67e519` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1535)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r959288158
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
Review Comment:
Also I wouldn't go too much into details here, and remove the mention about "can be represented as a struct array".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ksuarez1423 commented on a diff in pull request #14008: ARROW-13454: [C++][Docs] Tables vs Record Batches
Posted by GitBox <gi...@apache.org>.
ksuarez1423 commented on code in PR #14008:
URL: https://github.com/apache/arrow/pull/14008#discussion_r958917652
##########
docs/source/cpp/tables.rst:
##########
@@ -77,6 +77,17 @@ has a schema which must match its arrays' datatypes.
Record batches are a convenient unit of work for various serialization
and computation functions, possibly incremental.
+.. image:: tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a Record Batch, with
+ structure as described in text above.
+
+Because record batches can be represented as a struct array, they can be
+exported through the C data interface between implementations. Tables and
+chunked arrays, on the other hand, are concepts in the C++ implementation, not
Review Comment:
Knowledge of the C data interface aside, it may be worth being explicit where Tables can and cannot go -- for example, Python claims to have a Table class which is described similarly, and this seems like a good moment to be clear about incompatibilities/compatibilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org