You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sgilmore10 (via GitHub)" <gi...@apache.org> on 2023/07/11 13:47:51 UTC
[GitHub] [arrow] sgilmore10 opened a new pull request, #36615: GH-36614: [MATLAB] Store backing MATLAB array within C++ proxy objects instead of MATLAB classes
sgilmore10 opened a new pull request, #36615:
URL: https://github.com/apache/arrow/pull/36615
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
When building `arrow.array.<Numeric>Arrays` from native MATLAB arrays, we avoid copying by
Wrapping the MATLAB data inside a non-owning `arrow::Buffer`
Constructing an `arrow::ArrayData` object from the `arrow::Buffer`
Constructing the `arrow::Array` from the `arrow::Data` object.
Because the array's buffer does not own its backing data, we have been storing the original MATLAB array as a property called `MatlabArray` on the MATLAB `arrow.array.<Numeric>Arrays` classes.
This solution is not ideal because the backing MATLAB array is kept separate from the actual `std::shared_ptr<arrow::Array>`, which is stored within the C++ proxy objects (e.g. `arrow::matlab::array::proxy::NumericArray<float64>`). It would be much cleaner if the C++ proxy objects were also responsible for keeping the original MATLAB array alive.
### What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
1. Removed the `MatlabArray` property from the MATLAB class `matlab.io.NumericArray`
2. Added a private member variable called `mda_array` to `arrow::matlab::array::proxy::NumericArray<CType>` and `arrow::matlab::array::proxy::TimestampArray`. `mda_array` is a `matlab::dataArray` object. This member variable stores the MATLAB array that owns the memory the `arrow::Array` objects are backed by.
### Are these changes tested?
Existing tests used.
### Are there any user-facing changes?
No.
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sgilmore10 commented on pull request #36615: GH-36614: [MATLAB] Subclass arrow::Buffer to keep MATLAB data backing arrow::Arrays alive
Posted by "sgilmore10 (via GitHub)" <gi...@apache.org>.
sgilmore10 commented on PR #36615:
URL: https://github.com/apache/arrow/pull/36615#issuecomment-1632479312
> +1
>
> Could you update the PR title/description before we merge this?
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #36615: GH-36614: [MATLAB] Subclass arrow::Buffer to keep MATLAB data backing arrow::Arrays alive
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #36615:
URL: https://github.com/apache/arrow/pull/36615#issuecomment-1646680560
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit d8a336068540389196102b4e5366b82bad92987b.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/15261397883) has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sgilmore10 commented on pull request #36615: GH-36614: [MATLAB] Store backing MATLAB array within C++ proxy objects instead of MATLAB classes
Posted by "sgilmore10 (via GitHub)" <gi...@apache.org>.
sgilmore10 commented on PR #36615:
URL: https://github.com/apache/arrow/pull/36615#issuecomment-1631120370
Thanks for the suggestion @pitrou! That is definitely a better approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sgilmore10 commented on pull request #36615: GH-36614: [MATLAB] Store backing MATLAB array within C++ proxy objects instead of MATLAB classes
Posted by "sgilmore10 (via GitHub)" <gi...@apache.org>.
sgilmore10 commented on PR #36615:
URL: https://github.com/apache/arrow/pull/36615#issuecomment-1631337520
Based on @pitrou suggestion, I defined a `Buffer` subclass called `MatlabBuffer` that keeps the MATLAB data alive. Thanks again for the feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #36615: GH-36614: [MATLAB] Store backing MATLAB array within C++ proxy objects instead of MATLAB classes
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #36615:
URL: https://github.com/apache/arrow/pull/36615#issuecomment-1631013228
From the POV of Arrow C++, the idiomatic way to solve this issue would be to define your own `Buffer` subclass that keeps the required MATLAB object(s) alive. This is what we already do in PyArrow when we want to view a Numpy array as a PyArrow array:
https://github.com/apache/arrow/blob/c09b8c864b1e937a617a104a2b26644fa2f35dc3/python/pyarrow/src/arrow/python/numpy_convert.h#L42-L49
https://github.com/apache/arrow/blob/c09b8c864b1e937a617a104a2b26644fa2f35dc3/python/pyarrow/src/arrow/python/numpy_convert.cc#L40-L58
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney merged pull request #36615: GH-36614: [MATLAB] Subclass arrow::Buffer to keep MATLAB data backing arrow::Arrays alive
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney merged PR #36615:
URL: https://github.com/apache/arrow/pull/36615
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on a diff in pull request #36615: GH-36614: [MATLAB] Subclass arrow::Buffer to keep MATLAB data backing arrow::Arrays alive
Posted by "kevingurney (via GitHub)" <gi...@apache.org>.
kevingurney commented on code in PR #36615:
URL: https://github.com/apache/arrow/pull/36615#discussion_r1261647536
##########
matlab/src/cpp/arrow/matlab/array/proxy/timestamp_array.cc:
##########
@@ -20,24 +20,22 @@
#include "arrow/matlab/error/error.h"
#include "arrow/matlab/bit/pack.h"
#include "arrow/matlab/bit/unpack.h"
+#include "arrow/matlab/buffer/matlab_buffer.h"
#include "arrow/matlab/type/time_unit.h"
#include "arrow/util/utf8.h"
#include "arrow/type.h"
-#include "arrow/builder.h"
-
namespace arrow::matlab::array::proxy {
- namespace {
- const uint8_t* getUnpackedValidityBitmap(const ::matlab::data::TypedArray<bool>& valid_elements) {
- const auto valid_elements_iterator(valid_elements.cbegin());
- return reinterpret_cast<const uint8_t*>(valid_elements_iterator.operator->());
- }
- } // anonymous namespace
+ TimestampArray::TimestampArray(std::shared_ptr<arrow::TimestampArray> timestamp_array)
Review Comment:
Perhaps we should go with the convention of naming these input `Array` subclass constructor arguments `array` for consistency across the various subclasses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org