You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pw42020 (via GitHub)" <gi...@apache.org> on 2025/03/29 23:25:14 UTC
[PR] Remove MATLAB failing tutorial code in README [arrow]
pw42020 opened a new pull request, #45973:
URL: https://github.com/apache/arrow/pull/45973
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
Please remove this line and the above text before creating your pull request.
### Rationale for this change
MATLAB currently has multiple "example code" sections in the readme `matlab/doc/matlab_interface_for_apache_arrow_design.md` that have either been deprecated or were retrieved from other languages where MATLAB does not have the same endpoints. Example code that works out of the box not only helps with early developers get started using arrow, but also helps experienced developers ensure their development setup is in proper condition.
The broken endpoints described include:
#### Use Case 2
1. `arrow.Table(Var1, Var2, Var3)`
a. With the current implementation of arrow.Table, it cannot take multiple `arrow.array`s and create a table from that. All examples inside of `matlab/test/arrow/tabular/tTable.m` create a table by first creating a MATLAB table.
3. `arrow.FeatherTableWriter`
4. `arrow.FeatherTableReader`
5. `arrow.matlab2arrow`
#### Use Case 3
1. `importFromCDataInterface`
2. `ExportToCDataInterface`
3. `arrow.ipcwrite`
a. The current way to write ipc files using arrow, `arrow.io.ipc.RecordBatchFileWriter`, seems to be broken. The details are described in #45972.
### What changes are included in this PR?
I changed the use case README example code in use case 2 and use case 3 to use code that can be copy-and-pasted into MATLAB and work out of the box, rather than the current endpoints.
### Are these changes tested?
Yes. These changes have been tested by running each example piece of code inside of MATLAB inside of the changes and verifying that they work out of the box.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "pw42020 (via GitHub)" <gi...@apache.org>.
pw42020 commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2771019893
Hi @kevingurney thanks so much for your comments! I'm hoping to address all of these tomorrow. I appreciate your notes about how some of the changes I made could be updated to flow better in the existing README by changing some of the surrounding text. I'll be sure to incorporate that and the recommendations you made into my changes as well.
Thanks,
Pat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46048: [Docs][MATLAB] update README failing example code snippets [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784065648
:warning: GitHub issue #46048 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2779749411
Your changes look good!
@pw42020 - before I merge this, could you please update the description of this pull request to remove the note about the `RecordBatchFileWriter` being broken?
We generally try to ensure pull request descriptions are accurate and up to date before merging in changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Docs][MATLAB] update README failing example code snippets [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784966493
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 8dd6625b754921368c5c812e588d6215eddb25a1.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/40142348221) has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Docs][MATLAB] update README failing example code snippets [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784962747
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 8dd6625b754921368c5c812e588d6215eddb25a1.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/40142232682) has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "pw42020 (via GitHub)" <gi...@apache.org>.
pw42020 commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2779780643
Just did! Thanks so much for your help on this PR! Looking forward to helping out more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029336109
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
+>> writer.close();
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
###### Example Code:
``` matlab
->> featherTableReader = arrow.FeatherTableReader("data.feather");
+% open record batch file reader
+>> reader = arrow.io.ipc.RecordBatchFileReader(filename);
+
+% read in the first RecordBatch
+>> newBatch = reader.read(1);
->> AT = featherTableReader.read();
+% create a table that matches AT before it was written to data.arrow from the RecordBatch
+>> AT = table(newBatch);
```
#### Advanced MATLAB User Workflow for Implementing Support for Writing to Feather Files
Review Comment:
Marking this as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029334974
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -109,19 +109,7 @@ ans =
To serialize MATLAB data to a file on disk (e.g. Feather, Parquet), a MATLAB developer could start by constructing an `arrow.Table` using one of several different approaches.
-They could individually compose the table from a set of `arrow.Array` objects (one for each table variable).
-
-###### Example Code:
-``` matlab
->> Var1 = arrow.array(["foo"; "bar"; "baz"]);
-
->> Var2 = arrow.array([today; today + 1; today + 2]);
-
->> Var3 = arrow.array([10; 20; 30]);
-
->> AT = arrow.Table(Var1, Var2, Var3);
-```
-Alternatively, they could directly convert from an existing MATLAB `table` to an `arrow.Table` using a function like `arrow.matlab2arrow` to convert between an existing MATLAB `table` and an `arrow.Table`.
+They could directly convert from an existing MATLAB `table` to an `arrow.Table` using a function like `arrow.table`.
Review Comment:
Marking this as resolved for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029300769
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -131,33 +119,43 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> Density = [10.2; 20.5; 11.2; 13.7; 17.8];
->> T = table(Weight, Radius, Density); % Create a MATLAB table
+% Create a MATLAB `table`
+>> T = table(Weight, Radius, Density);
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+% Create an `arrow.tabular.Table` from the MATLAB `table`
+>> AT = arrow.table(T);
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.internal.io.feather.Writer`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
-
->> featherTableWriter.write(AT, "data.feather");
+% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = "data.arrow";
Review Comment:
To make it clear that this is Feather V1 and *NOT* Feather **V2** (i.e. the Arrow IPC format), I think it would be better to use the extension `.feather` here.
```suggestion
>> filename = "data.feather";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "pw42020 (via GitHub)" <gi...@apache.org>.
pw42020 commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2778584756
Hi @kevingurney, thanks again for your time reviewing my changes. I took the vast majority of the changes you recommended, except for ones in regard to updating the current feather example surrounding documentation to IPC to match my then-current supporting IPC code.
I actually hadn't realized the IPC format was not synonymous with the feather format, and changed all of the examples that use feather back to feather with working code. Let me know if there's any other changes you'd like me to make!
Pat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Docs][MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784083959
Actually, my apologies. I realize that this is technically a "MINOR PR" because it only consists of documentation changes to a single file.
In this case - the title of the PR should follow the form: `MINOR: [${COMPONENT}] ${SUMMARY}`.
I've updated the PR title accordingly, and will close the associated issue I opened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Docs][MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784088582
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029311841
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,65 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
+
+Since we require multiple lines to import our matlab Arrow array into python, we'll use a function called `pyrunfile`, which allows us to [execute Python statements in a file supplied as an argument to the function](https://www.mathworks.com/help/matlab/ref/pyrunfile.html).
###### Example Code:
+
+```python
+# file located in same directory as the matlab file, named import_from_c.py
+import pyarrow as pa
+array = pa.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress)
+```
+
``` matlab
% Create a MATLAB arrow.Array.
>> AA = arrow.array([1, 2, 3, 4, 5]);
+% Export C Data Interface C-style structs for `arrow.array.Array` values and schema
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
+
% Export the MATLAB arrow.Array to the C Data Interface format, returning the
% memory addresses of the required ArrowArray and ArrowSchema C-style structs.
->> [arrayMemoryAddress, schemaMemoryAddress] = AA.exportToCDataInterface();
+>> AA.export(cArray.Address, cSchema.Address);
% Import the memory addresses of the C Data Interface format structs to create a pyarrow.Array.
->> PA = py.pyarrow.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress);
+>> PA = pyrunfile("import_from_c.py", "array", arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
```
Conversely, a user can create an Arrow array using PyArrow and share it with MATLAB. To do this, they can call the method `_export_to_c` to export a `pyarrow.Array` to the C Data Interface format.
-The memory addresses to the `ArrowArray` and `ArrowSchema` structs populated by the call to `_export_to_c` can be passed to the static method `arrow.Array.importFromCDataInterface` to construct a MATLAB `arrow.Array` with zero copies.
+**NOTE:** Since the python calls to `_export_to_c` and `_import_from_c` have underscores at the beginning of their names, they cannot be called directly in MATLAB. MATLAB member functions or variables are [not allowed to start with an underscore](https://www.mathworks.com/help/matlab/matlab_prog/variable-names.html).
-The example code below is adapted from the [`test_cffi.py` test cases for PyArrow].
+To initialize a Python `pyarrow` array, the MATLAB `pyrunfile` command can be used, which supports the execution of Python code containing variables and functions with names that start with an underscore.
Review Comment:
```suggestion
To initialize a Python `pyarrow` array, `pyrunfile` can (again) be used to execute a Python script containing variables and functions with names that start with an underscore.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029342822
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,67 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses for the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
+
+Multiple lines of Python are required to import the Arrow array from MATLAB. Therefore, the function [`pyrunfile`]((https://www.mathworks.com/help/matlab/ref/pyrunfile.html)) can be used when can run Python scripts defined in an external file.
Review Comment:
```suggestion
Multiple lines of Python are required to import the Arrow array from MATLAB. Therefore, the function [`pyrunfile`]((https://www.mathworks.com/help/matlab/ref/pyrunfile.html)) can be used which can run Python scripts defined in an external file.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029335846
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
+>> writer.close();
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
Review Comment:
Marking as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029342822
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,67 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses for the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
+
+Multiple lines of Python are required to import the Arrow array from MATLAB. Therefore, the function [`pyrunfile`]((https://www.mathworks.com/help/matlab/ref/pyrunfile.html)) can be used when can run Python scripts defined in an external file.
Review Comment:
Sorry, I missed this typo before:
```suggestion
Multiple lines of Python are required to import the Arrow array from MATLAB. Therefore, the function [`pyrunfile`]((https://www.mathworks.com/help/matlab/ref/pyrunfile.html)) can be used which can run Python scripts defined in an external file.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784057952
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2765003403
Thank you very much for the PR, @pw42020!
As you pointed out, these design docs are quite old and have gone very stale. The use cases were initially intended to provide a "theoretical" preview of what the MATLAB interface might look like in the future, but the API has not evolved exactly as planned over time.
I want to make sure that I provide a thorough review since these changes are likely to have a significant impact on new users of the MATLAB interface (as you rightfully pointed out), so I will take a closer look at this over the next day or so
My apologies for the delayed review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029335577
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -205,21 +212,18 @@ The example code below is adapted from the [`test_cffi.py` test cases for PyArro
>> PA = py.pyarrow.array([1, 2, 3, 4, 5]);
% Create ArrowArray and ArrowSchema C-style structs adhering to the Arrow C Data Interface format.
->> array = py.pyarrow.cffi.ffi.new("struct ArrowArray*")
-
->> arrayMemoryAddress = py.int(py.pyarrow.cffi.ffi.cast("uintptr_t", array));
-
->> schema = py.pyarrow.cffi.ffi.new("struct ArrowSchema*")
-
->> schemaMemoryAddress = py.int(py.pyarrow.cffi.ffi.cast("uintptr_t", schema));
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
% Export the pyarrow.Array to the C Data Interface format, populating the required ArrowArray and ArrowShema structs.
->> PA.export_to_c(arrayMemoryAddress, schemaMemoryAddress)
+>> pyrun("import pyarrow as pa; PA._export_to_c(arrayMemoryAddress, schemaMemoryAddress)", PA=PA, arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
Review Comment:
Marking as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Docs][MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney merged PR #45973:
URL: https://github.com/apache/arrow/pull/45973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029301782
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -131,33 +119,43 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> Density = [10.2; 20.5; 11.2; 13.7; 17.8];
->> T = table(Weight, Radius, Density); % Create a MATLAB table
+% Create a MATLAB `table`
+>> T = table(Weight, Radius, Density);
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+% Create an `arrow.tabular.Table` from the MATLAB `table`
+>> AT = arrow.table(T);
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.internal.io.feather.Writer`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
-
->> featherTableWriter.write(AT, "data.feather");
+% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = "data.arrow";
+% Write the `arrow.tabular.RecordBatch` using Feather locally to `data.arrow`
+>> writer = arrow.internal.io.feather.Writer(filename);
+>> writer.write(recordBatch);
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.internal.io.feather.Reader`.
Review Comment:
```suggestion
The Feather V1 file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB, the user could instantiate an `arrow.internal.io.feather.Reader`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2023129894
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -109,19 +109,7 @@ ans =
To serialize MATLAB data to a file on disk (e.g. Feather, Parquet), a MATLAB developer could start by constructing an `arrow.Table` using one of several different approaches.
-They could individually compose the table from a set of `arrow.Array` objects (one for each table variable).
-
-###### Example Code:
-``` matlab
->> Var1 = arrow.array(["foo"; "bar"; "baz"]);
-
->> Var2 = arrow.array([today; today + 1; today + 2]);
-
->> Var3 = arrow.array([10; 20; 30]);
-
->> AT = arrow.Table(Var1, Var2, Var3);
-```
-Alternatively, they could directly convert from an existing MATLAB `table` to an `arrow.Table` using a function like `arrow.matlab2arrow` to convert between an existing MATLAB `table` and an `arrow.Table`.
+They could directly convert from an existing MATLAB `table` to an `arrow.Table` using a function like `arrow.table`.
Review Comment:
The title of Use Case 2 is "Developer Workflow for Writing a MATLAB Table to a Feather File". However, the changes you've made are using the Arrow IPC file I/O APIs, rather than Feather V1 APIs. If we want to change this to talk about the Arrow IPC format, then we should probably replace all mentions of Feather with "Arrow IPC file format", or something similar.
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
Review Comment:
```suggestion
>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,22 +180,28 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
###### Example Code:
``` matlab
% Create a MATLAB arrow.Array.
>> AA = arrow.array([1, 2, 3, 4, 5]);
+% create C Data Interface for array values and schema
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
+
% Export the MATLAB arrow.Array to the C Data Interface format, returning the
% memory addresses of the required ArrowArray and ArrowSchema C-style structs.
->> [arrayMemoryAddress, schemaMemoryAddress] = AA.exportToCDataInterface();
+>> AA.export(cArray.Address, cSchema.Address);
% Import the memory addresses of the C Data Interface format structs to create a pyarrow.Array.
->> PA = py.pyarrow.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress);
+>> PA = pyrun("import pyarrow as pa; array = pa.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress)", "array", arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
```
Conversely, a user can create an Arrow array using PyArrow and share it with MATLAB. To do this, they can call the method `_export_to_c` to export a `pyarrow.Array` to the C Data Interface format.
+Since the python calls to `_export_to_c` and `_import_from_c` have underscores at the beginning, they cannot be called in MATLAB. MATLAB member functions or variables are not allowed to start with an underscore shown [in the "variable names" syntax page on the Mathworks website](https://www.mathworks.com/help/matlab/matlab_prog/variable-names.html). Instead, to initialize your Python pyarrow array, you must use MATLAB's built-in `pyrun` to execute code in Python, which allows for variables and functions to start with underscore.
Review Comment:
```suggestion
**NOTE:** Since the python calls to `_export_to_c` and `_import_from_c` have underscores at the beginning of their names, they cannot be called directly in MATLAB. MATLAB member functions or variables are [not allowed to start with an underscore](https://www.mathworks.com/help/matlab/matlab_prog/variable-names.html).
To initialize a Python `pyarrow` array, the MATLAB `pyrunfile` command can be used, which supports the execution of Python code containing variables and functions with names that start with an underscore.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
+>> writer.close();
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
Review Comment:
```suggestion
The Arrow IPC file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -109,19 +109,7 @@ ans =
To serialize MATLAB data to a file on disk (e.g. Feather, Parquet), a MATLAB developer could start by constructing an `arrow.Table` using one of several different approaches.
-They could individually compose the table from a set of `arrow.Array` objects (one for each table variable).
-
-###### Example Code:
-``` matlab
->> Var1 = arrow.array(["foo"; "bar"; "baz"]);
-
->> Var2 = arrow.array([today; today + 1; today + 2]);
-
->> Var3 = arrow.array([10; 20; 30]);
-
->> AT = arrow.Table(Var1, Var2, Var3);
-```
-Alternatively, they could directly convert from an existing MATLAB `table` to an `arrow.Table` using a function like `arrow.matlab2arrow` to convert between an existing MATLAB `table` and an `arrow.Table`.
+They could directly convert from an existing MATLAB `table` to an `arrow.Table` using a function like `arrow.table`.
Review Comment:
```suggestion
They could directly convert from an existing MATLAB `table` to an `arrow.tabular.Table` using a function like `arrow.table`.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
+>> writer.close();
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
###### Example Code:
``` matlab
->> featherTableReader = arrow.FeatherTableReader("data.feather");
+% open record batch file reader
+>> reader = arrow.io.ipc.RecordBatchFileReader(filename);
+
+% read in the first RecordBatch
+>> newBatch = reader.read(1);
->> AT = featherTableReader.read();
+% create a table that matches AT before it was written to data.arrow from the RecordBatch
+>> AT = table(newBatch);
```
#### Advanced MATLAB User Workflow for Implementing Support for Writing to Feather Files
Review Comment:
Since there is potentially a lot of content to update here for this to be cohesive, maybe we can do this in a separate, follow up pull request, but we should probably rewrite this section a bit to be more accurate.
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,22 +180,28 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
###### Example Code:
``` matlab
% Create a MATLAB arrow.Array.
>> AA = arrow.array([1, 2, 3, 4, 5]);
+% create C Data Interface for array values and schema
Review Comment:
```suggestion
% Export C Data Interface C-style structs for `arrow.array.Array` values and schema
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
Review Comment:
```suggestion
% Create an `arrow.tabular.Table` from the MATLAB `table`
>> AT = arrow.table(T);
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
Review Comment:
```suggestion
% Create a MATLAB `table`
>> T = table(Weight, Radius, Density);
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
+>> writer.close();
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
###### Example Code:
``` matlab
->> featherTableReader = arrow.FeatherTableReader("data.feather");
+% open record batch file reader
Review Comment:
```suggestion
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,22 +180,28 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
###### Example Code:
``` matlab
% Create a MATLAB arrow.Array.
>> AA = arrow.array([1, 2, 3, 4, 5]);
+% create C Data Interface for array values and schema
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
+
% Export the MATLAB arrow.Array to the C Data Interface format, returning the
% memory addresses of the required ArrowArray and ArrowSchema C-style structs.
->> [arrayMemoryAddress, schemaMemoryAddress] = AA.exportToCDataInterface();
+>> AA.export(cArray.Address, cSchema.Address);
% Import the memory addresses of the C Data Interface format structs to create a pyarrow.Array.
->> PA = py.pyarrow.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress);
+>> PA = pyrun("import pyarrow as pa; array = pa.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress)", "array", arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
Review Comment:
This is a bit hard to parse since it's all on one line. Maybe we could break this out into a separate code block and use `pyrunfile` instead of `pyrun` here?
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
Review Comment:
For simplicitly, I think we can just a relative path `"data.arrow"` here, unless there is some issue with the code supporting that (in which case, we should create an issue for that).
```suggestion
>> filename = "data.arrow";
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
+>> writer.close();
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
###### Example Code:
``` matlab
->> featherTableReader = arrow.FeatherTableReader("data.feather");
+% open record batch file reader
+>> reader = arrow.io.ipc.RecordBatchFileReader(filename);
+
+% read in the first RecordBatch
+>> newBatch = reader.read(1);
->> AT = featherTableReader.read();
+% create a table that matches AT before it was written to data.arrow from the RecordBatch
Review Comment:
```suggestion
% Create a MATLAB `table` from the `arrow.tabular.RecordBatch`
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
+>> writer.close();
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.io.ipc.RecordBatchFileReader`.
###### Example Code:
``` matlab
->> featherTableReader = arrow.FeatherTableReader("data.feather");
+% open record batch file reader
+>> reader = arrow.io.ipc.RecordBatchFileReader(filename);
+
+% read in the first RecordBatch
Review Comment:
```suggestion
% Read in the first RecordBatch
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
Review Comment:
```suggestion
% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -133,23 +121,36 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> T = table(Weight, Radius, Density); % Create a MATLAB table
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+>> AT = arrow.table(T); % Create an arrow.Table
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.io.ipc.RecordBatchFileWriter`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
+% create a RecordBatch out of the table from the previous example
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = fullfile(pwd, "data.arrow");
->> featherTableWriter.write(AT, "data.feather");
+% write the RecordBatch schema and the RecordBatch itself
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+>> writer.writeRecordBatch(recordBatch);
+
+% close the writer to enable reading
Review Comment:
```suggestion
% Close the writer to finalize writing
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -205,21 +212,18 @@ The example code below is adapted from the [`test_cffi.py` test cases for PyArro
>> PA = py.pyarrow.array([1, 2, 3, 4, 5]);
% Create ArrowArray and ArrowSchema C-style structs adhering to the Arrow C Data Interface format.
->> array = py.pyarrow.cffi.ffi.new("struct ArrowArray*")
-
->> arrayMemoryAddress = py.int(py.pyarrow.cffi.ffi.cast("uintptr_t", array));
-
->> schema = py.pyarrow.cffi.ffi.new("struct ArrowSchema*")
-
->> schemaMemoryAddress = py.int(py.pyarrow.cffi.ffi.cast("uintptr_t", schema));
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
% Export the pyarrow.Array to the C Data Interface format, populating the required ArrowArray and ArrowShema structs.
->> PA.export_to_c(arrayMemoryAddress, schemaMemoryAddress)
+>> pyrun("import pyarrow as pa; PA._export_to_c(arrayMemoryAddress, schemaMemoryAddress)", PA=PA, arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
Review Comment:
Same comment as above, perhaps, we could use `pyrunfile` here instead?
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -205,21 +212,18 @@ The example code below is adapted from the [`test_cffi.py` test cases for PyArro
>> PA = py.pyarrow.array([1, 2, 3, 4, 5]);
% Create ArrowArray and ArrowSchema C-style structs adhering to the Arrow C Data Interface format.
->> array = py.pyarrow.cffi.ffi.new("struct ArrowArray*")
-
->> arrayMemoryAddress = py.int(py.pyarrow.cffi.ffi.cast("uintptr_t", array));
-
->> schema = py.pyarrow.cffi.ffi.new("struct ArrowSchema*")
-
->> schemaMemoryAddress = py.int(py.pyarrow.cffi.ffi.cast("uintptr_t", schema));
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
% Export the pyarrow.Array to the C Data Interface format, populating the required ArrowArray and ArrowShema structs.
->> PA.export_to_c(arrayMemoryAddress, schemaMemoryAddress)
+>> pyrun("import pyarrow as pa; PA._export_to_c(arrayMemoryAddress, schemaMemoryAddress)", PA=PA, arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
% Import the C Data Interface structs to create a MATLAB arrow.Array.
->> AA = arrow.Array.importFromCDataInterface(arrayMemoryAddress, schemaMemoryAddress);
+>> AA = arrow.array.Array.import(cArray, cSchema);
```
+For the same reasons as mentioned above in the export from MATLAB to Python example, we must also run the export command inside of the `pyrun` function.
Review Comment:
I think we can omit this comment since it has already been explained above.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029299904
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -131,33 +119,43 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> Density = [10.2; 20.5; 11.2; 13.7; 17.8];
->> T = table(Weight, Radius, Density); % Create a MATLAB table
+% Create a MATLAB `table`
+>> T = table(Weight, Radius, Density);
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+% Create an `arrow.tabular.Table` from the MATLAB `table`
+>> AT = arrow.table(T);
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.internal.io.feather.Writer`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
-
->> featherTableWriter.write(AT, "data.feather");
+% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = "data.arrow";
+% Write the `arrow.tabular.RecordBatch` using Feather locally to `data.arrow`
Review Comment:
```suggestion
% Write the `arrow.tabular.RecordBatch` to disk as a Feather V1 file named `data.feather`
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -131,33 +119,43 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> Density = [10.2; 20.5; 11.2; 13.7; 17.8];
->> T = table(Weight, Radius, Density); % Create a MATLAB table
+% Create a MATLAB `table`
+>> T = table(Weight, Radius, Density);
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+% Create an `arrow.tabular.Table` from the MATLAB `table`
+>> AT = arrow.table(T);
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.internal.io.feather.Writer`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
-
->> featherTableWriter.write(AT, "data.feather");
+% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = "data.arrow";
Review Comment:
To make it clear that this is Feather V1 and *NOT* Feather V2 (i.e. the Arrow IPC format), I think it would be better to use the extension `.feather` here.
```suggestion
>> filename = "data.feather";
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,65 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
+
+Since we require multiple lines to import our matlab Arrow array into python, we'll use a function called `pyrunfile`, which allows us to [execute Python statements in a file supplied as an argument to the function](https://www.mathworks.com/help/matlab/ref/pyrunfile.html).
Review Comment:
```suggestion
Multiple lines of Python are required to import the Arrow array from MATLAB. Therefore, the function [`pyrunfile`]((https://www.mathworks.com/help/matlab/ref/pyrunfile.html)) can be used when can run Python scripts defined in an external file.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,65 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
+
+Since we require multiple lines to import our matlab Arrow array into python, we'll use a function called `pyrunfile`, which allows us to [execute Python statements in a file supplied as an argument to the function](https://www.mathworks.com/help/matlab/ref/pyrunfile.html).
###### Example Code:
+
+```python
+# file located in same directory as the matlab file, named import_from_c.py
+import pyarrow as pa
+array = pa.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress)
+```
+
``` matlab
% Create a MATLAB arrow.Array.
>> AA = arrow.array([1, 2, 3, 4, 5]);
+% Export C Data Interface C-style structs for `arrow.array.Array` values and schema
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
+
% Export the MATLAB arrow.Array to the C Data Interface format, returning the
% memory addresses of the required ArrowArray and ArrowSchema C-style structs.
->> [arrayMemoryAddress, schemaMemoryAddress] = AA.exportToCDataInterface();
+>> AA.export(cArray.Address, cSchema.Address);
% Import the memory addresses of the C Data Interface format structs to create a pyarrow.Array.
->> PA = py.pyarrow.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress);
+>> PA = pyrunfile("import_from_c.py", "array", arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
```
Conversely, a user can create an Arrow array using PyArrow and share it with MATLAB. To do this, they can call the method `_export_to_c` to export a `pyarrow.Array` to the C Data Interface format.
-The memory addresses to the `ArrowArray` and `ArrowSchema` structs populated by the call to `_export_to_c` can be passed to the static method `arrow.Array.importFromCDataInterface` to construct a MATLAB `arrow.Array` with zero copies.
+**NOTE:** Since the python calls to `_export_to_c` and `_import_from_c` have underscores at the beginning of their names, they cannot be called directly in MATLAB. MATLAB member functions or variables are [not allowed to start with an underscore](https://www.mathworks.com/help/matlab/matlab_prog/variable-names.html).
-The example code below is adapted from the [`test_cffi.py` test cases for PyArrow].
+To initialize a Python `pyarrow` array, the MATLAB `pyrunfile` command can be used, which supports the execution of Python code containing variables and functions with names that start with an underscore.
+
+The memory addresses to the `ArrowArray` and `ArrowSchema` structs populated by the call to `_export_to_c` can be passed to the static method `arrow.Array.importFromCDataInterface` to construct a MATLAB `arrow.Array` with zero copies.
###### Example Code:
+
+```python
+# file located in same directory as the matlab file, named export_to_c.py
Review Comment:
```suggestion
# Filename: export_to_c.py
# Note: This file is located in same directory as the MATLAB file.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -131,33 +119,43 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> Density = [10.2; 20.5; 11.2; 13.7; 17.8];
->> T = table(Weight, Radius, Density); % Create a MATLAB table
+% Create a MATLAB `table`
+>> T = table(Weight, Radius, Density);
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+% Create an `arrow.tabular.Table` from the MATLAB `table`
+>> AT = arrow.table(T);
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.internal.io.feather.Writer`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
-
->> featherTableWriter.write(AT, "data.feather");
+% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = "data.arrow";
+% Write the `arrow.tabular.RecordBatch` using Feather locally to `data.arrow`
+>> writer = arrow.internal.io.feather.Writer(filename);
+>> writer.write(recordBatch);
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.internal.io.feather.Reader`.
###### Example Code:
``` matlab
->> featherTableReader = arrow.FeatherTableReader("data.feather");
+>> reader = arrow.internal.io.feather.Reader(filename);
->> AT = featherTableReader.read();
+% Read in the first RecordBatch
+>> newBatch = reader.read();
+
+% Create a MATLAB `table` from the `arrow.tabular.RecordBatch`
+>> AT = table(newBatch);
```
#### Advanced MATLAB User Workflow for Implementing Support for Writing to Feather Files
-To add support for writing to Feather files, an advanced MATLAB user could use the MATLAB and C++ APIs offered by the MATLAB Interface for Apache Arrow to create `arrow.FeatherTableWriter`.
+To add support for writing to Feather files, an advanced MATLAB user could use the MATLAB and C++ APIs offered by the MATLAB Interface for Apache Arrow to create `arrow.internal.io.feather.Writer`.
Review Comment:
```suggestion
To add support for writing to Feather V1 files, an advanced MATLAB user could use the MATLAB and C++ APIs offered by the MATLAB Interface for Apache Arrow to create `arrow.internal.io.feather.Writer`.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,65 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
Review Comment:
```suggestion
Memory addresses for the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -131,33 +119,43 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> Density = [10.2; 20.5; 11.2; 13.7; 17.8];
->> T = table(Weight, Radius, Density); % Create a MATLAB table
+% Create a MATLAB `table`
+>> T = table(Weight, Radius, Density);
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+% Create an `arrow.tabular.Table` from the MATLAB `table`
+>> AT = arrow.table(T);
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.internal.io.feather.Writer`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
-
->> featherTableWriter.write(AT, "data.feather");
+% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = "data.arrow";
+% Write the `arrow.tabular.RecordBatch` using Feather locally to `data.arrow`
+>> writer = arrow.internal.io.feather.Writer(filename);
+>> writer.write(recordBatch);
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.internal.io.feather.Reader`.
Review Comment:
```suggestion
The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB, the user could instantiate an `arrow.internal.io.feather.Reader`.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -131,33 +119,43 @@ Alternatively, they could directly convert from an existing MATLAB `table` to an
>> Density = [10.2; 20.5; 11.2; 13.7; 17.8];
->> T = table(Weight, Radius, Density); % Create a MATLAB table
+% Create a MATLAB `table`
+>> T = table(Weight, Radius, Density);
->> AT = arrow.matlab2arrow(T); % Create an arrow.Table
+% Create an `arrow.tabular.Table` from the MATLAB `table`
+>> AT = arrow.table(T);
```
-To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.FeatherTableWriter`.
+
+To serialize the `arrow.Table`, `AT`, to a file (e.g. Feather) on disk, the user could then instantiate an `arrow.internal.io.feather.Writer`.
###### Example Code:
``` matlab
->> featherTableWriter = arrow.FeatherTableWriter();
-
->> featherTableWriter.write(AT, "data.feather");
+% Make an `arrow.tabular.RecordBatch` from the `arrow.tabular.Table` created in the previous step
+>> recordBatch = arrow.recordBatch(AT);
+>> filename = "data.arrow";
+% Write the `arrow.tabular.RecordBatch` using Feather locally to `data.arrow`
+>> writer = arrow.internal.io.feather.Writer(filename);
+>> writer.write(recordBatch);
```
-The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.FeatherTableReader`.
+The Feather file could then be read and operated on by an external process like Rust or Go. To read it back into MATLAB after modification by another process, the user could instantiate an `arrow.internal.io.feather.Reader`.
###### Example Code:
``` matlab
->> featherTableReader = arrow.FeatherTableReader("data.feather");
+>> reader = arrow.internal.io.feather.Reader(filename);
->> AT = featherTableReader.read();
+% Read in the first RecordBatch
+>> newBatch = reader.read();
+
+% Create a MATLAB `table` from the `arrow.tabular.RecordBatch`
+>> AT = table(newBatch);
```
#### Advanced MATLAB User Workflow for Implementing Support for Writing to Feather Files
-To add support for writing to Feather files, an advanced MATLAB user could use the MATLAB and C++ APIs offered by the MATLAB Interface for Apache Arrow to create `arrow.FeatherTableWriter`.
+To add support for writing to Feather files, an advanced MATLAB user could use the MATLAB and C++ APIs offered by the MATLAB Interface for Apache Arrow to create `arrow.internal.io.feather.Writer`.
They would need to author a [MEX function] (e.g. `featherwriteMEX`), which can be called directly by MATLAB code. Within their MEX function, they could use `arrow::matlab::unwrap_table` to convert between the MATLAB representation of the Arrow memory (`arrow.Table`) and the equivalent C++ representation (`arrow::Table`). Once the `arrow.Table` has been "unwrapped" into a C++ `arrow::Table`, it can be passed to the appropriate Arrow C++ library API for writing to a Feather file (`arrow::ipc::feather::WriteTable`).
-An analogous workflow could be followed to create `arrow.FeatherTableReader` to enable reading from Feather files.
+An analogous workflow could be followed to create `arrow.internal.io.feather.Reader` to enable reading from Feather files.
Review Comment:
```suggestion
An analogous workflow could be followed to create `arrow.internal.io.feather.Reader` to enable reading from Feather V1 files.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,65 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
+
+Since we require multiple lines to import our matlab Arrow array into python, we'll use a function called `pyrunfile`, which allows us to [execute Python statements in a file supplied as an argument to the function](https://www.mathworks.com/help/matlab/ref/pyrunfile.html).
###### Example Code:
+
+```python
+# file located in same directory as the matlab file, named import_from_c.py
Review Comment:
```suggestion
# Filename: import_from_c.py
# Note: This file is located in same directory as the MATLAB file.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,47 +177,65 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
+
+Since we require multiple lines to import our matlab Arrow array into python, we'll use a function called `pyrunfile`, which allows us to [execute Python statements in a file supplied as an argument to the function](https://www.mathworks.com/help/matlab/ref/pyrunfile.html).
###### Example Code:
+
+```python
+# file located in same directory as the matlab file, named import_from_c.py
+import pyarrow as pa
+array = pa.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress)
+```
+
``` matlab
% Create a MATLAB arrow.Array.
>> AA = arrow.array([1, 2, 3, 4, 5]);
+% Export C Data Interface C-style structs for `arrow.array.Array` values and schema
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
+
% Export the MATLAB arrow.Array to the C Data Interface format, returning the
% memory addresses of the required ArrowArray and ArrowSchema C-style structs.
->> [arrayMemoryAddress, schemaMemoryAddress] = AA.exportToCDataInterface();
+>> AA.export(cArray.Address, cSchema.Address);
% Import the memory addresses of the C Data Interface format structs to create a pyarrow.Array.
->> PA = py.pyarrow.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress);
+>> PA = pyrunfile("import_from_c.py", "array", arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
```
Conversely, a user can create an Arrow array using PyArrow and share it with MATLAB. To do this, they can call the method `_export_to_c` to export a `pyarrow.Array` to the C Data Interface format.
-The memory addresses to the `ArrowArray` and `ArrowSchema` structs populated by the call to `_export_to_c` can be passed to the static method `arrow.Array.importFromCDataInterface` to construct a MATLAB `arrow.Array` with zero copies.
+**NOTE:** Since the python calls to `_export_to_c` and `_import_from_c` have underscores at the beginning of their names, they cannot be called directly in MATLAB. MATLAB member functions or variables are [not allowed to start with an underscore](https://www.mathworks.com/help/matlab/matlab_prog/variable-names.html).
-The example code below is adapted from the [`test_cffi.py` test cases for PyArrow].
+To initialize a Python `pyarrow` array, the MATLAB `pyrunfile` command can be used, which supports the execution of Python code containing variables and functions with names that start with an underscore.
Review Comment:
```suggestion
To initialize a Python `pyarrow` array, `pyrunfile` can (again) be used to execute a Python script containing containing variables and functions with names that start with an underscore.
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -240,7 +256,18 @@ For large tables used in a multi-process "data processing pipeline", a user coul
>> AT = arrow.Table(Var1, Var2, Var3);
% Write the MATLAB arrow.Table to the Arrow IPC File Format on disk.
->> arrow.ipcwrite(AT, "data.arrow");
+>> recordBatch = arrow.recordBatch(AT);
+
+>> filename = "data.arrow"
+
+% open `data.arrow` as an IPC file
Review Comment:
```suggestion
% Open `data.arrow` as an IPC file
```
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -240,7 +256,18 @@ For large tables used in a multi-process "data processing pipeline", a user coul
>> AT = arrow.Table(Var1, Var2, Var3);
% Write the MATLAB arrow.Table to the Arrow IPC File Format on disk.
->> arrow.ipcwrite(AT, "data.arrow");
+>> recordBatch = arrow.recordBatch(AT);
+
+>> filename = "data.arrow"
+
+% open `data.arrow` as an IPC file
+>> writer = arrow.io.ipc.RecordBatchFileWriter(filename, recordBatch.Schema);
+
+% write the `RecordBatch` to `data.arrow`
Review Comment:
```suggestion
% Write the `RecordBatch` to `data.arrow`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #45973:
URL: https://github.com/apache/arrow/pull/45973#discussion_r2029335367
##########
matlab/doc/matlab_interface_for_apache_arrow_design.md:
##########
@@ -179,22 +180,28 @@ Roughly speaking, local memory sharing workflows can be divided into two categor
To share a MATLAB `arrow.Array` with PyArrow efficiently, a user could use the `exportToCDataInterface` method to export the Arrow memory wrapped by an `arrow.Array` to the C Data Interface format, consisting of two C-style structs, [`ArrowArray`] and [`ArrowSchema`], which represent the Arrow data and associated metadata.
-Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `exportToCDataInterface`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `py.pyarrow.Array._import_from_c`.
+Memory addresses to the `ArrowArray` and `ArrowSchema` structs are returned by the call to `export`. These addresses can be passed to Python directly, without having to make any copies of the underlying Arrow data structures that they refer to. A user can then wrap the underlying data pointed to by the `ArrowArray` struct (which is already in the [Arrow Columnar Format]), as well as extract the necessary metadata from the `ArrowSchema` struct, to create a `pyarrow.Array` by using the static method `pyarrow.Array._import_from_c`.
###### Example Code:
``` matlab
% Create a MATLAB arrow.Array.
>> AA = arrow.array([1, 2, 3, 4, 5]);
+% create C Data Interface for array values and schema
+>> cArray = arrow.c.Array();
+>> cSchema = arrow.c.Schema();
+
% Export the MATLAB arrow.Array to the C Data Interface format, returning the
% memory addresses of the required ArrowArray and ArrowSchema C-style structs.
->> [arrayMemoryAddress, schemaMemoryAddress] = AA.exportToCDataInterface();
+>> AA.export(cArray.Address, cSchema.Address);
% Import the memory addresses of the C Data Interface format structs to create a pyarrow.Array.
->> PA = py.pyarrow.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress);
+>> PA = pyrun("import pyarrow as pa; array = pa.Array._import_from_c(arrayMemoryAddress, schemaMemoryAddress)", "array", arrayMemoryAddress=cArray.Address, schemaMemoryAddress=cSchema.Address);
Review Comment:
The latest changes use `pyrunfile`. Marking this as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove MATLAB failing tutorial code in README [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2764287785
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46048: [Docs][MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784072743
@pw42020 - FYI, its typically best practice to create a specific GitHub issue for each PR. Also, the format for the title of PRs that the project generally follows is: `GH-<issue-id>: <issue-title>`. This format is actually enforced programmatically by the project's PR merge script.
I went ahead and made a new [issue](https://github.com/apache/arrow/issues/46048) and updated the PR title, accordingly, so that we can get this merged in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MATLAB] update README failing example code snippets [arrow]
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on PR #45973:
URL: https://github.com/apache/arrow/pull/45973#issuecomment-2784057833
Thanks so much @pw42020!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org