You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2019/06/11 17:55:55 UTC
[arrow] branch master updated: ARROW-3897: [MATLAB] Add MATLAB
support for writing numeric datatypes to a Feather file
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new f5045c9 ARROW-3897: [MATLAB] Add MATLAB support for writing numeric datatypes to a Feather file
f5045c9 is described below
commit f5045c9d9d344a1b9fd95286b86ce2f9e20a5759
Author: Kevin Gurney <ke...@gmail.com>
AuthorDate: Tue Jun 11 12:55:42 2019 -0500
ARROW-3897: [MATLAB] Add MATLAB support for writing numeric datatypes to a Feather file
**Writing Feather Files**
- Currently the MATLAB interface to Feather supports reading numeric datatypes (`double`, `single`, `uint*` and `int*`) from Feather files using the `featherread` function.
- This pull request adds a `featherwrite` function to serialize a MATLAB `table` containing numeric datatypes to a Feather file.
**Validity (Null) Bitmap Support**
- Previously, there was a `TODO` to add support for interpreting validity (null) bitmaps when reading from a Feather file.
- This pull request adds support for reading missing values from Feather files. Because there is no built in representation for missing values of integer type in MATLAB, `featherread` will cast integer columns with missing values to `double` and set missing entries to `NaN`.
**Testing**
- Removed binary Feather files used for testing and replaced them with temporary files generated during test execution by `featherwrite`.
- Added additional test cases to improve code coverage.
**Notes**
- A preliminary code review was performed in https://github.com/mathworks/arrow/pull/6. I saw the discussion on https://github.com/apache/arrow/pull/4321 about conducting code reviews directly in apache/arrow following https://apache.org/theapacheway/. I'll be sure to do code reviews directly in apache/arrow moving forward.
- This pull request contains a lot of changes. My apologies for this. Our intent was to refactor the reading and writing code and improve code coverage, but we should have isolated these changes into separate patches/JIRA issues. We'll be more thoughtful about breaking our submissions into small, single purpose patches in the future.
- This pull request includes contributions from @rdmello and @sihuiliu. Thanks!
Author: Kevin Gurney <ke...@gmail.com>
Author: rdmello <ry...@mathworks.com>
Author: Sihui <si...@mathworks.com>
Closes #4328 from kevingurney/ARROW-3897 and squashes the following commits:
5e5686537 <rdmello> Use mxLogical instead of uint8_t in bit-packing/unpacking operations.
ef3288da8 <rdmello> Use uint8_t* instead of bool* in bit-packing and unpacking code.
35143c22c <rdmello> Use BitmapReader in VisitBits to improve performance.
baaaf5291 <rdmello> Avoid MSVC build failures by removing implicit conversions of std::array<T,N>::iterator to uint8_t*
d029ac802 <rdmello> Resolve formatting issue in bit-util-test.cc.
d57c338f9 <Kevin Gurney> Remove use of const qualifiers with primitive type function arguments.
a796bbfa9 <Kevin Gurney> Remove unnecessary explicit use of arrow::.
74668da4a <Kevin Gurney> Replace mlarrow namespace with arrow::matlab namespace.
ff8d220e6 <rdmello> Use AllocateResizableBuffer in order to handle allocation issues better
2e4f7ddb7 <Kevin Gurney> Remove use of Feather file version information in Feather reading and writing code.
6d311bcf6 <rdmello> Use GenerateBitsUnrolled for better performance when bit-packing validity bitmap
0105f69f0 <rdmello> Address lint errors and code review feedback.
16ead0a5c <rdmello> Add VisitBits and VisitBitsUnrolled to read from a bitmap.
ce8628b62 <Kevin Gurney> Remove unnecessary use of arrow::Status lvalue in FeatherWriter::WriteVariables. Remove unnecessary use of ARROW_RETURN_NOT_OK in FeatherWriter::Open.
39d4707c9 <Kevin Gurney> Remove unnecessary arrow::Array lvalue in WriteNumericData.
0f11c9790 <Kevin Gurney> Use std::make_shared in WriteNumericData.
684f283b6 <Kevin Gurney> Return nullptr from WriteVariableData.
3d6cb2c44 <Kevin Gurney> Change FeatherWriter::WriteMetadata return type to void.
63757e491 <Kevin Gurney> Use RETURN_NOT_OK in FeatherReader::Open.
a027dd616 <Kevin Gurney> Refactor featherwrite MATLAB table conversion code into utility function mlarrow.util.table2mlarrow.
845394a85 <Kevin Gurney> Refactor invalid MATLAB table variable name handling code into utility mlarrow.util.makeValidMATLABTableVariableNames. Add test case for handling invalid MATLAB table variable names.
53c55fb35 <rdmello> Adding unicode conversion utility for column names and refactoring bit-packing/unpacking code
ce3faad5f <Kevin Gurney> Move createMetadataStruct and createVariableStruct into mlarrow.util package for use in featherwrite code.
2d6a1f275 <Kevin Gurney> Factor out negation used in unpacked validity bitmap array check.
b35303263 <Kevin Gurney> Break up long line in NumericDatatypesWithNaNRow test case into multiple lines for improved readability.
0454b13ab <Kevin Gurney> Replace all uses of typedef with using.
80cbce34f <Kevin Gurney> Clean up and refactor tfeather.m. Add tfeathermex.m numeric nulls round trip test. Create test utilities directory.
bed64e15c <Kevin Gurney> Modify tfeather.m to make all tests fail if MEX files are not on the MATLAB path
3cdb22197 <rdmello> Fixing FEATHERWRITE test failure for 0-by-n inputs.
4eeb11a53 <Sihui> Added featherwrite tests, removed binary files and modified old tests to roundtrip.
55ac75260 <rdmello> Adding support for writing the nulls (validity) bitmap for numeric types to a Feather file from MATLAB
5ef69316a <Kevin Gurney> Implement validity (null) bitmap support for featherread numeric types
e92a21159 <Kevin Gurney> Update README.md to include example code for writing a MATLAB table to a Feather file
268222027 <Kevin Gurney> Implement featherwrite MATLAB code for numeric datatypes support
c86a1d36d <rdmello> Add support for writing numeric datatypes to Feather files.
---
cpp/src/arrow/util/bit-util-test.cc | 119 ++++++++
cpp/src/arrow/util/bit-util.h | 63 ++++
matlab/CMakeLists.txt | 9 +
matlab/README.md | 16 +-
matlab/src/+mlarrow/+util/createMetadataStruct.m | 24 ++
matlab/src/+mlarrow/+util/createVariableStruct.m | 24 ++
.../+util/makeValidMATLABTableVariableNames.m | 42 +++
matlab/src/+mlarrow/+util/table2mlarrow.m | 83 +++++
matlab/src/feather_reader.cc | 228 +++++++++-----
matlab/src/feather_reader.h | 48 +--
matlab/src/feather_writer.cc | 338 +++++++++++++++++++++
matlab/src/feather_writer.h | 75 +++++
matlab/src/featherread.m | 45 ++-
matlab/src/featherreadmex.cc | 5 +-
matlab/src/featherwrite.m | 44 +++
.../src/{featherreadmex.cc => featherwritemex.cc} | 17 +-
matlab/src/matlab_traits.h | 52 ++--
matlab/src/util/handle_status.cc | 41 +--
matlab/src/util/handle_status.h | 14 +-
matlab/src/util/unicode_conversion.cc | 63 ++++
.../util/{handle_status.h => unicode_conversion.h} | 22 +-
matlab/test/corrupted_feather_file.feather | 5 -
matlab/test/not_a_feather_file.feather | 18 --
...ic_datatypes_6th_variable_name_is_empty.feather | Bin 976 -> 0 bytes
.../test/numeric_datatypes_with_nan_column.feather | Bin 1040 -> 0 bytes
matlab/test/numeric_datatypes_with_nan_row.feather | Bin 1272 -> 0 bytes
.../test/numeric_datatypes_with_no_nulls.feather | Bin 984 -> 0 bytes
matlab/test/tfeather.m | 232 ++++++++++++++
matlab/test/tfeathermex.m | 76 +++++
matlab/test/tfeatherread.m | 141 ---------
matlab/test/util/createTable.m | 68 +++++
.../test/util/createVariablesAndMetadataStructs.m | 98 ++++++
matlab/test/util/featherMEXRoundTrip.m | 22 ++
matlab/test/util/featherRoundTrip.m | 22 ++
34 files changed, 1695 insertions(+), 359 deletions(-)
diff --git a/cpp/src/arrow/util/bit-util-test.cc b/cpp/src/arrow/util/bit-util-test.cc
index 20b1a92..af70e22 100644
--- a/cpp/src/arrow/util/bit-util-test.cc
+++ b/cpp/src/arrow/util/bit-util-test.cc
@@ -15,12 +15,15 @@
// specific language governing permissions and limitations
// under the License.
+#include <algorithm>
+#include <array>
#include <climits>
#include <cstdint>
#include <cstring>
#include <functional>
#include <limits>
#include <memory>
+#include <string>
#include <vector>
#include <gtest/gtest.h>
@@ -375,6 +378,122 @@ TYPED_TEST(TestGenerateBits, NormalOperation) {
}
}
+// Tests for VisitBits and VisitBitsUnrolled. Based on the tests for GenerateBits and
+// GenerateBitsUnrolled.
+struct VisitBitsFunctor {
+ void operator()(const uint8_t* bitmap, int64_t start_offset, int64_t length,
+ bool* destination) {
+ auto writer = [&](const bool& bit_value) { *destination++ = bit_value; };
+ return internal::VisitBits(bitmap, start_offset, length, writer);
+ }
+};
+
+struct VisitBitsUnrolledFunctor {
+ void operator()(const uint8_t* bitmap, int64_t start_offset, int64_t length,
+ bool* destination) {
+ auto writer = [&](const bool& bit_value) { *destination++ = bit_value; };
+ return internal::VisitBitsUnrolled(bitmap, start_offset, length, writer);
+ }
+};
+
+/* Define a typed test class with some utility members. */
+template <typename T>
+class TestVisitBits : public ::testing::Test {
+ protected:
+ // The bitmap size that will be used throughout the VisitBits tests.
+ static const int64_t kBitmapSizeInBytes = 32;
+
+ // Typedefs for the source and expected destination types in this test.
+ using PackedBitmapType = std::array<uint8_t, kBitmapSizeInBytes>;
+ using UnpackedBitmapType = std::array<bool, 8 * kBitmapSizeInBytes>;
+
+ // Helper functions to generate the source bitmap and expected destination
+ // arrays.
+ static PackedBitmapType generate_packed_bitmap() {
+ PackedBitmapType bitmap;
+ // Assign random values into the source array.
+ random_bytes(kBitmapSizeInBytes, 0, bitmap.data());
+ return bitmap;
+ }
+
+ static UnpackedBitmapType generate_unpacked_bitmap(PackedBitmapType bitmap) {
+ // Use a BitmapReader (tested earlier) to populate the expected
+ // unpacked bitmap.
+ UnpackedBitmapType result;
+ internal::BitmapReader reader(bitmap.data(), 0, 8 * kBitmapSizeInBytes);
+ for (int64_t index = 0; index < 8 * kBitmapSizeInBytes; ++index) {
+ result[index] = reader.IsSet();
+ reader.Next();
+ }
+ return result;
+ }
+
+ // A pre-defined packed bitmap for use in test cases.
+ const PackedBitmapType packed_bitmap_;
+
+ // The expected unpacked bitmap that would be generated if each bit in
+ // the entire source bitmap was correctly unpacked to bytes.
+ const UnpackedBitmapType expected_unpacked_bitmap_;
+
+ // Define a test constructor that populates the packed bitmap and the expected
+ // unpacked bitmap.
+ TestVisitBits()
+ : packed_bitmap_(generate_packed_bitmap()),
+ expected_unpacked_bitmap_(generate_unpacked_bitmap(packed_bitmap_)) {}
+};
+
+using VisitBitsTestTypes = ::testing::Types<VisitBitsFunctor, VisitBitsUnrolledFunctor>;
+TYPED_TEST_CASE(TestVisitBits, VisitBitsTestTypes);
+
+/* Test bit-unpacking when reading less than eight bits from the input */
+TYPED_TEST(TestVisitBits, NormalOperation) {
+ typename TestFixture::UnpackedBitmapType unpacked_bitmap;
+ const int64_t start_offsets[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 21, 31, 32};
+ const int64_t lengths[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 16,
+ 17, 21, 31, 32, 100, 201, 202, 203, 204, 205, 206, 207};
+ const bool fill_values[] = {false, true};
+
+ for (const bool fill_value : fill_values) {
+ auto is_unmodified = [=](bool value) -> bool { return value == fill_value; };
+
+ for (const int64_t start_offset : start_offsets) {
+ for (const int64_t length : lengths) {
+ std::string failure_info = std::string("fill value: ") +
+ std::to_string(fill_value) +
+ ", start offset: " + std::to_string(start_offset) +
+ ", length: " + std::to_string(length);
+ // Pre-fill the unpacked_bitmap array.
+ unpacked_bitmap.fill(fill_value);
+
+ // Attempt to read bits from the input bitmap into the unpacked_bitmap bitmap.
+ using VisitBitsFunctor = TypeParam;
+ VisitBitsFunctor()(this->packed_bitmap_.data(), start_offset, length,
+ unpacked_bitmap.data() + start_offset);
+
+ // Verify that the correct values have been written in the [start_offset,
+ // start_offset+length) range.
+ EXPECT_TRUE(std::equal(unpacked_bitmap.begin() + start_offset,
+ unpacked_bitmap.begin() + start_offset + length,
+ this->expected_unpacked_bitmap_.begin() + start_offset))
+ << "Invalid bytes unpacked when using " << failure_info;
+
+ // Verify that the unpacked_bitmap array has not changed before or after
+ // the [start_offset, start_offset+length) range.
+ EXPECT_TRUE(std::all_of(unpacked_bitmap.begin(),
+ unpacked_bitmap.begin() + start_offset, is_unmodified))
+ << "Unexpected modification to unpacked_bitmap array before written range "
+ "when using "
+ << failure_info;
+ EXPECT_TRUE(std::all_of(unpacked_bitmap.begin() + start_offset + length,
+ unpacked_bitmap.end(), is_unmodified))
+ << "Unexpected modification to unpacked_bitmap array after written range "
+ "when using "
+ << failure_info;
+ }
+ }
+ }
+}
+
struct BitmapOperation {
virtual Status Call(MemoryPool* pool, const uint8_t* left, int64_t left_offset,
const uint8_t* right, int64_t right_offset, int64_t length,
diff --git a/cpp/src/arrow/util/bit-util.h b/cpp/src/arrow/util/bit-util.h
index b7de112..4742271 100644
--- a/cpp/src/arrow/util/bit-util.h
+++ b/cpp/src/arrow/util/bit-util.h
@@ -53,6 +53,7 @@
#define ARROW_BYTE_SWAP32 __builtin_bswap32
#endif
+#include <cmath>
#include <cstdint>
#include <cstring>
#include <limits>
@@ -402,6 +403,9 @@ static inline bool GetBit(const uint8_t* bits, uint64_t i) {
return (bits[i >> 3] >> (i & 0x07)) & 1;
}
+// Gets the i-th bit from a byte. Should only be used with i <= 7.
+static inline bool GetBitFromByte(uint8_t byte, uint8_t i) { return byte & kBitmask[i]; }
+
static inline void ClearBit(uint8_t* bits, int64_t i) {
bits[i / 8] &= kFlippedBitmask[i % 8];
}
@@ -686,6 +690,65 @@ void GenerateBitsUnrolled(uint8_t* bitmap, int64_t start_offset, int64_t length,
}
}
+// A function that visits each bit in a bitmap and calls a visitor function with a
+// boolean representation of that bit. This is intended to be analogous to
+// GenerateBits.
+template <class Visitor>
+void VisitBits(const uint8_t* bitmap, int64_t start_offset, int64_t length,
+ Visitor&& visit) {
+ BitmapReader reader(bitmap, start_offset, length);
+ for (int64_t index = 0; index < length; ++index) {
+ visit(reader.IsSet());
+ reader.Next();
+ }
+}
+
+// Like VisitBits(), but unrolls its main loop for better performance.
+template <class Visitor>
+void VisitBitsUnrolled(const uint8_t* bitmap, int64_t start_offset, int64_t length,
+ Visitor&& visit) {
+ if (length == 0) {
+ return;
+ }
+
+ // Start by visiting any bits preceding the first full byte.
+ int64_t num_bits_before_full_bytes =
+ BitUtil::RoundUpToMultipleOf8(start_offset) - start_offset;
+ // Truncate num_bits_before_full_bytes if it is greater than length.
+ if (num_bits_before_full_bytes > length) {
+ num_bits_before_full_bytes = length;
+ }
+ // Use the non loop-unrolled VisitBits since we don't want to add branches
+ VisitBits<Visitor>(bitmap, start_offset, num_bits_before_full_bytes, visit);
+
+ // Shift the start pointer to the first full byte and compute the
+ // number of full bytes to be read.
+ const uint8_t* first_full_byte = bitmap + BitUtil::CeilDiv(start_offset, 8);
+ const int64_t num_full_bytes = (length - num_bits_before_full_bytes) / 8;
+
+ // Iterate over each full byte of the input bitmap and call the visitor in
+ // a loop-unrolled manner.
+ for (int64_t byte_index = 0; byte_index < num_full_bytes; ++byte_index) {
+ // Get the current bit-packed byte value from the bitmap.
+ const uint8_t byte = *(first_full_byte + byte_index);
+
+ // Execute the visitor function on each bit of the current byte.
+ visit(BitUtil::GetBitFromByte(byte, 0));
+ visit(BitUtil::GetBitFromByte(byte, 1));
+ visit(BitUtil::GetBitFromByte(byte, 2));
+ visit(BitUtil::GetBitFromByte(byte, 3));
+ visit(BitUtil::GetBitFromByte(byte, 4));
+ visit(BitUtil::GetBitFromByte(byte, 5));
+ visit(BitUtil::GetBitFromByte(byte, 6));
+ visit(BitUtil::GetBitFromByte(byte, 7));
+ }
+
+ // Write any leftover bits in the last byte.
+ const int64_t num_bits_after_full_bytes = (length - num_bits_before_full_bytes) % 8;
+ VisitBits<Visitor>(first_full_byte + num_full_bytes, 0, num_bits_after_full_bytes,
+ visit);
+}
+
// ----------------------------------------------------------------------
// Bitmap utilities
diff --git a/matlab/CMakeLists.txt b/matlab/CMakeLists.txt
index 3165a58..37e28c6 100755
--- a/matlab/CMakeLists.txt
+++ b/matlab/CMakeLists.txt
@@ -42,5 +42,14 @@ matlab_add_mex(NAME featherreadmex
SRC src/featherreadmex.cc
src/feather_reader.cc
src/util/handle_status.cc
+ src/util/unicode_conversion.cc
LINK_TO ${ARROW_SHARED_LIB})
target_include_directories(featherreadmex PRIVATE ${ARROW_INCLUDE_DIR})
+
+# Build featherwrite mex file based on the arrow shared library
+matlab_add_mex(NAME featherwritemex
+ SRC src/featherwritemex.cc
+ src/feather_writer.cc
+ src/util/handle_status.cc
+ LINK_TO ${ARROW_SHARED_LIB})
+target_include_directories(featherwritemex PRIVATE ${ARROW_INCLUDE_DIR})
diff --git a/matlab/README.md b/matlab/README.md
index 5edf08b..edf991e 100644
--- a/matlab/README.md
+++ b/matlab/README.md
@@ -23,7 +23,7 @@
This is a very early stage MATLAB interface to the Apache Arrow C++ libraries.
-The current code only supports reading numeric types from Feather files.
+The current code only supports reading/writing numeric types from/to Feather files.
## Building from source
@@ -86,14 +86,20 @@ Run the `test` function to execute the unit tests:
>> addpath build;
```
-### Read a Feather file into MATLAB as a table
+### Write a MATLAB table to a Feather file
``` matlab
->> filename = fullfile('arrow', 'matlab', 'test', 'numericDatatypesWithNoNulls.feather');
->> t = featherread(filename);
+>> t = array2table(rand(10, 10));
+>> filename = 'table.feather';
+>> featherwrite(filename,t);
```
-This should return a MATLAB table datatype containing the Feather file contents.
+### Read a Feather file into a MATLAB table
+
+``` matlab
+>> filename = 'table.feather';
+>> t = featherread(filename);
+```
## Running the tests
diff --git a/matlab/src/+mlarrow/+util/createMetadataStruct.m b/matlab/src/+mlarrow/+util/createMetadataStruct.m
new file mode 100644
index 0000000..7a23970
--- /dev/null
+++ b/matlab/src/+mlarrow/+util/createMetadataStruct.m
@@ -0,0 +1,24 @@
+function metadata = createMetadataStruct(description, numRows, numVariables)
+% CREATEMETADATASTRUCT Helper function for creating Feather MEX metadata
+% struct.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+metadata = struct('Description', description, ...
+ 'NumRows', numRows, ...
+ 'NumVariables', numVariables);
+end
+
diff --git a/matlab/src/+mlarrow/+util/createVariableStruct.m b/matlab/src/+mlarrow/+util/createVariableStruct.m
new file mode 100644
index 0000000..99f52d8
--- /dev/null
+++ b/matlab/src/+mlarrow/+util/createVariableStruct.m
@@ -0,0 +1,24 @@
+function variable = createVariableStruct(type, data, valid, name)
+% CREATEVARIABLESTRUCT Helper function for creating Feather MEX variable
+% struct.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+variable = struct('Type', type, ...
+ 'Data', data, ...
+ 'Valid', valid, ...
+ 'Name', name);
+end
\ No newline at end of file
diff --git a/matlab/src/+mlarrow/+util/makeValidMATLABTableVariableNames.m b/matlab/src/+mlarrow/+util/makeValidMATLABTableVariableNames.m
new file mode 100644
index 0000000..ba5e072
--- /dev/null
+++ b/matlab/src/+mlarrow/+util/makeValidMATLABTableVariableNames.m
@@ -0,0 +1,42 @@
+function [variableNames, variableDescriptions] = makeValidMATLABTableVariableNames(columnNames)
+% makeValidMATLABTableVariableNames Makes valid MATLAB table variable names
+% from a set of Feather table column names.
+%
+% [variableNames, variableDescriptions] = makeValidMATLABTableVariableNames(columnNames)
+% Modifies the input Feather table columnNames to be valid MATLAB table
+% variable names if they are not already. If any of the Feather table columnNames
+% are invalid MATLAB table variable names, then the original columnNames are returned
+% in variableDescriptions to be stored in the table.Properties.VariableDescriptions
+% property.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+ variableNames = string(columnNames);
+ variableDescriptions = strings(0, 0);
+
+ validVariableNames = false(1, length(variableNames));
+ for ii = 1:length(variableNames)
+ validVariableNames(ii) = isvarname(variableNames(ii));
+ end
+
+ if ~all(validVariableNames)
+ variableDescriptions = strings(1, length(columnNames));
+ variableDescriptions(validVariableNames) = "";
+ variableDescriptions(~validVariableNames) = compose("Original variable name: '%s'", ...
+ variableNames(~validVariableNames));
+ variableNames(~validVariableNames) = matlab.lang.makeValidName(variableNames(~validVariableNames));
+ end
+end
diff --git a/matlab/src/+mlarrow/+util/table2mlarrow.m b/matlab/src/+mlarrow/+util/table2mlarrow.m
new file mode 100644
index 0000000..3103724
--- /dev/null
+++ b/matlab/src/+mlarrow/+util/table2mlarrow.m
@@ -0,0 +1,83 @@
+function [variables, metadata] = table2mlarrow(t)
+%TABLE2MLARROW Converts a MATLAB table into a form
+% suitable for passing to the mlarrow C++ MEX layer.
+%
+% [VARIABLES, METADATA] = TABLE2MLARROW(T)
+% Takes a MATLAB table T and returns struct array equivalents
+% which are suitable for passing to the mlarrow C++ MEX layer.
+%
+% VARIABLES is an 1xN struct array representing the the table variables.
+%
+% VARIABLES contains the following fields:
+%
+% Field Name Class Description
+% ------------ ------- ----------------------------------------------
+% Name char Variable's name
+% Type char Variable's MATLAB datatype
+% Data numeric Variable's data
+% Valid logical 0 = invalid (null), 1 = valid (non-null) value
+%
+% METADATA is a 1x1 struct array with the following fields:
+%
+% METADATA contains the following fields:
+%
+% Field Name Class Description
+% ------------ ------- ----------------------------------------------
+% Description char Table description (T.Properties.Description)
+% NumRows double Number of table rows (height(T))
+% NumVariables double Number of table variables (width(T))
+%
+% See also FEATHERREAD, FEATHERWRITE.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+import mlarrow.util.*;
+
+% Struct array representing the underlying data of each variable
+% in the given table.
+variables = repmat(createVariableStruct('', [], [], ''), 1, width(t));
+
+% Struct representing table-level metadata.
+metadata = createMetadataStruct(t.Properties.Description, height(t), width(t));
+
+% Iterate over each variable in the given table,
+% extracting the underlying array data.
+for ii = 1:width(t)
+ data = t.(ii);
+ % Multi-column table variables are unsupported.
+ if ~isvector(data)
+ error('MATLAB:arrow:MultiColumnVariablesUnsupported', ...
+ 'Multi-column table variables are unsupported by featherwrite.');
+ end
+ % Get the datatype of the current variable's underlying array.
+ variables(ii).Type = class(data);
+ % Break the datatype down into its constituent components, if appropriate.
+ switch variables(ii).Type
+ % For numeric variables, the underlying array data can
+ % be passed to the C++ layer directly.
+ case {'uint8', 'uint16', 'uint32', 'uint64', ...
+ 'int8', 'int16', 'int32', 'int64', ...
+ 'single', 'double'}
+ variables(ii).Data = data;
+ otherwise
+ error('MATLAB:arrow:UnsupportedVariableType', ...
+ ['Type ' variables(ii).Type ' is unsupported by featherwrite.']);
+ end
+ variables(ii).Valid = ~ismissing(data);
+ variables(ii).Name = t.Properties.VariableNames{ii};
+end
+
+end
diff --git a/matlab/src/feather_reader.cc b/matlab/src/feather_reader.cc
index 22bbc66..1c1b21c 100644
--- a/matlab/src/feather_reader.cc
+++ b/matlab/src/feather_reader.cc
@@ -15,34 +15,39 @@
// specific language governing permissions and limitations
// under the License.
+#include <algorithm>
+#include <cmath>
+
#include <arrow/io/file.h>
#include <arrow/ipc/feather.h>
#include <arrow/status.h>
#include <arrow/table.h>
#include <arrow/type.h>
+#include <arrow/util/bit-util.h>
#include <mex.h>
#include "feather_reader.h"
#include "matlab_traits.h"
#include "util/handle_status.h"
+#include "util/unicode_conversion.h"
-namespace mlarrow {
-
+namespace arrow {
+namespace matlab {
namespace internal {
// Read the name of variable i from the Feather file as a mxArray*.
-mxArray* ReadVariableName(const std::shared_ptr<arrow::Column>& column) {
- return mxCreateString(column->name().c_str());
+mxArray* ReadVariableName(const std::shared_ptr<Column>& column) {
+ return matlab::util::ConvertUTF8StringToUTF16CharMatrix(column->name());
}
template <typename ArrowDataType>
-mxArray* ReadNumericVariableData(const std::shared_ptr<arrow::Column>& column) {
- typedef typename MatlabTraits<ArrowDataType>::MatlabType MatlabType;
- typedef typename arrow::TypeTraits<ArrowDataType>::ArrayType ArrowArrayType;
+mxArray* ReadNumericVariableData(const std::shared_ptr<Column>& column) {
+ using MatlabType = typename MatlabTraits<ArrowDataType>::MatlabType;
+ using ArrowArrayType = typename TypeTraits<ArrowDataType>::ArrayType;
- std::shared_ptr<arrow::ChunkedArray> chunked_array = column->data();
- const int num_chunks = chunked_array->num_chunks();
+ std::shared_ptr<ChunkedArray> chunked_array = column->data();
+ const int32_t num_chunks = chunked_array->num_chunks();
const mxClassID matlab_class_id = MatlabTraits<ArrowDataType>::matlab_class_id;
// Allocate a numeric mxArray* with the correct mxClassID based on the type of the
@@ -52,13 +57,18 @@ mxArray* ReadNumericVariableData(const std::shared_ptr<arrow::Column>& column) {
int64_t mx_array_offset = 0;
// Iterate over each arrow::Array in the arrow::ChunkedArray.
- for (int i = 0; i < num_chunks; ++i) {
- std::shared_ptr<arrow::Array> array = chunked_array->chunk(i);
+ for (int32_t i = 0; i < num_chunks; ++i) {
+ std::shared_ptr<Array> array = chunked_array->chunk(i);
const int64_t chunk_length = array->length();
- std::shared_ptr<ArrowArrayType> arr = std::static_pointer_cast<ArrowArrayType>(array);
- const auto data = arr->raw_values();
- MatlabType* dt = MatlabTraits<ArrowDataType>::GetData(variable_data);
- std::copy(data, data + chunk_length, dt + mx_array_offset);
+ std::shared_ptr<ArrowArrayType> integer_array = std::static_pointer_cast<ArrowArrayType>(array);
+
+ // Get a raw pointer to the Arrow array data.
+ const MatlabType* source = integer_array->raw_values();
+
+ // Get a mutable pointer to the MATLAB array data and std::copy the
+ // Arrow array data into it.
+ MatlabType* destination = MatlabTraits<ArrowDataType>::GetData(variable_data);
+ std::copy(source, source + chunk_length, destination + mx_array_offset);
mx_array_offset += chunk_length;
}
@@ -66,31 +76,30 @@ mxArray* ReadNumericVariableData(const std::shared_ptr<arrow::Column>& column) {
}
// Read the data of variable i from the Feather file as a mxArray*.
-mxArray* ReadVariableData(const std::shared_ptr<arrow::Column>& column) {
- std::shared_ptr<arrow::DataType> type = column->type();
+mxArray* ReadVariableData(const std::shared_ptr<Column>& column) {
+ std::shared_ptr<DataType> type = column->type();
switch (type->id()) {
- case arrow::Type::FLOAT:
- return ReadNumericVariableData<arrow::FloatType>(column);
- case arrow::Type::DOUBLE:
- return ReadNumericVariableData<arrow::DoubleType>(column);
- case arrow::Type::UINT8:
- return ReadNumericVariableData<arrow::UInt8Type>(column);
- case arrow::Type::UINT16:
- return ReadNumericVariableData<arrow::UInt16Type>(column);
- case arrow::Type::UINT32:
- return ReadNumericVariableData<arrow::UInt32Type>(column);
- case arrow::Type::UINT64:
- return ReadNumericVariableData<arrow::UInt64Type>(column);
- case arrow::Type::INT8:
- return ReadNumericVariableData<arrow::Int8Type>(column);
- case arrow::Type::INT16:
- return ReadNumericVariableData<arrow::Int16Type>(column);
- case arrow::Type::INT32:
- return ReadNumericVariableData<arrow::Int32Type>(column);
- case arrow::Type::INT64:
- return ReadNumericVariableData<arrow::Int64Type>(column);
-
+ case Type::FLOAT:
+ return ReadNumericVariableData<FloatType>(column);
+ case Type::DOUBLE:
+ return ReadNumericVariableData<DoubleType>(column);
+ case Type::UINT8:
+ return ReadNumericVariableData<UInt8Type>(column);
+ case Type::UINT16:
+ return ReadNumericVariableData<UInt16Type>(column);
+ case Type::UINT32:
+ return ReadNumericVariableData<UInt32Type>(column);
+ case Type::UINT64:
+ return ReadNumericVariableData<UInt64Type>(column);
+ case Type::INT8:
+ return ReadNumericVariableData<Int8Type>(column);
+ case Type::INT16:
+ return ReadNumericVariableData<Int16Type>(column);
+ case Type::INT32:
+ return ReadNumericVariableData<Int32Type>(column);
+ case Type::INT64:
+ return ReadNumericVariableData<Int64Type>(column);
default: {
mexErrMsgIdAndTxt("MATLAB:arrow:UnsupportedArrowType",
"Unsupported arrow::Type '%s' for variable '%s'",
@@ -102,40 +111,111 @@ mxArray* ReadVariableData(const std::shared_ptr<arrow::Column>& column) {
return nullptr;
}
-// Read the nulls of variable i from the Feather file as a mxArray*.
-mxArray* ReadVariableNulls(const std::shared_ptr<arrow::Column>& column) {
- // TODO: Implement proper null value support. For the time being,
- // we will simply return a zero initialized logical array to MATLAB.
- return mxCreateLogicalMatrix(column->length(), 1);
+// arrow::Buffers are bit-packed, while mxLogical arrays aren't. This utility
+// uses an Arrow utility to copy each bit of an arrow::Buffer into each byte
+// of an mxLogical array.
+void BitUnpackBuffer(const std::shared_ptr<Buffer>& source, int64_t length,
+ mxLogical* destination) {
+ const uint8_t* source_data = source->data();
+
+ // Call into an Arrow utility to visit each bit in the bitmap.
+ auto visitFcn = [&](mxLogical is_valid) { *destination++ = is_valid; };
+
+ const int64_t start_offset = 0;
+ arrow::internal::VisitBitsUnrolled(source_data, start_offset, length, visitFcn);
+}
+
+// Populates the validity bitmap from an arrow::Array or an arrow::Column,
+// writes to a zero-initialized destination buffer.
+// Implements a fast path for the fully-valid and fully-invalid cases.
+// Returns true if the destination buffer was successfully populated.
+template <typename ArrowType>
+bool TryBitUnpackFastPath(const std::shared_ptr<ArrowType>& array, mxLogical* destination) {
+ const int64_t null_count = array->null_count();
+ const int64_t length = array->length();
+
+ if (null_count == length) {
+ // The source array/column is filled with invalid values. Since mxCreateLogicalMatrix
+ // zero-initializes the destination buffer, we can return without changing anything
+ // in the destination buffer.
+ return true;
+ } else if (null_count == 0) {
+ // The source array/column contains only valid values. Fill the destination buffer
+ // with 'true'.
+ std::fill(destination, destination + length, true);
+ return true;
+ }
+
+ // Return false to indicate that we couldn't fill the entire validity bitmap.
+ return false;
+}
+
+// Read the validity (null) bitmap of variable i from the Feather
+// file as an mxArray*.
+mxArray* ReadVariableValidityBitmap(const std::shared_ptr<Column>& column) {
+ // Allocate an mxLogical array to store the validity (null) bitmap values.
+ // Note: All Arrow arrays can have an associated validity (null) bitmap.
+ // The Apache Arrow specification defines 0 (false) to represent an
+ // invalid (null) array entry and 1 (true) to represent a valid
+ // (non-null) array entry.
+ mxArray* validity_bitmap = mxCreateLogicalMatrix(column->length(), 1);
+ mxLogical* validity_bitmap_unpacked = mxGetLogicals(validity_bitmap);
+
+ // The Apache Arrow specification allows validity (null) bitmaps
+ // to be unallocated if there are no null values. In this case,
+ // we simply return a logical array filled with the value true.
+ if (TryBitUnpackFastPath(column, validity_bitmap_unpacked)) {
+ // Return early since the validity bitmap was already filled.
+ return validity_bitmap;
+ }
+
+ std::shared_ptr<ChunkedArray> chunked_array = column->data();
+ const int32_t num_chunks = chunked_array->num_chunks();
+
+ int64_t mx_array_offset = 0;
+ // Iterate over each arrow::Array in the arrow::ChunkedArray.
+ for (int32_t chunk_index = 0; chunk_index < num_chunks; ++chunk_index) {
+ std::shared_ptr<Array> array = chunked_array->chunk(chunk_index);
+ const int64_t array_length = array->length();
+
+ if (!TryBitUnpackFastPath(array, validity_bitmap_unpacked + mx_array_offset)) {
+ // Couldn't fill the full validity bitmap at once. Call an optimized loop-unrolled
+ // implementation instead that goes byte-by-byte and populates the validity bitmap.
+ BitUnpackBuffer(array->null_bitmap(), array_length,
+ validity_bitmap_unpacked + mx_array_offset);
+ }
+
+ mx_array_offset += array_length;
+ }
+
+ return validity_bitmap;
}
-// Read the type of variable i from the Feather file as a mxArray*.
-mxArray* ReadVariableType(const std::shared_ptr<arrow::Column>& column) {
- return mxCreateString(column->type()->name().c_str());
+// Read the type name of an Arrow column as an mxChar array.
+mxArray* ReadVariableType(const std::shared_ptr<Column>& column) {
+ return util::ConvertUTF8StringToUTF16CharMatrix(column->type()->name());
}
-// MATLAB arrays cannot be larger than 2^48.
+// MATLAB arrays cannot be larger than 2^48 elements.
static constexpr uint64_t MAX_MATLAB_SIZE = static_cast<uint64_t>(0x01) << 48;
} // namespace internal
-arrow::Status FeatherReader::Open(const std::string& filename,
- std::shared_ptr<FeatherReader>* feather_reader) {
+Status FeatherReader::Open(const std::string& filename,
+ std::shared_ptr<FeatherReader>* feather_reader) {
*feather_reader = std::shared_ptr<FeatherReader>(new FeatherReader());
+
// Open file with given filename as a ReadableFile.
- std::shared_ptr<arrow::io::ReadableFile> readable_file(nullptr);
- auto status = arrow::io::ReadableFile::Open(filename, &readable_file);
- if (!status.ok()) {
- return status;
- }
+ std::shared_ptr<io::ReadableFile> readable_file(nullptr);
+
+ RETURN_NOT_OK(io::ReadableFile::Open(filename, &readable_file));
+
// TableReader expects a RandomAccessFile.
- std::shared_ptr<arrow::io::RandomAccessFile> random_access_file(readable_file);
+ std::shared_ptr<io::RandomAccessFile> random_access_file(readable_file);
+
// Open the Feather file for reading with a TableReader.
- status = arrow::ipc::feather::TableReader::Open(random_access_file,
- &(*feather_reader)->table_reader_);
- if (!status.ok()) {
- return status;
- }
+ RETURN_NOT_OK(ipc::feather::TableReader::Open(
+ random_access_file, &(*feather_reader)->table_reader_));
// Read the table metadata from the Feather file.
(*feather_reader)->num_rows_ = (*feather_reader)->table_reader_->num_rows();
@@ -144,7 +224,6 @@ arrow::Status FeatherReader::Open(const std::string& filename,
(*feather_reader)->table_reader_->HasDescription()
? (*feather_reader)->table_reader_->GetDescription()
: "";
- (*feather_reader)->version_ = (*feather_reader)->table_reader_->version();
if ((*feather_reader)->num_rows_ > internal::MAX_MATLAB_SIZE ||
(*feather_reader)->num_variables_ > internal::MAX_MATLAB_SIZE) {
@@ -153,13 +232,13 @@ arrow::Status FeatherReader::Open(const std::string& filename,
(*feather_reader)->num_rows_, (*feather_reader)->num_variables_);
}
- return status;
+ return Status::OK();
}
// Read the table metadata from the Feather file as a mxArray*.
mxArray* FeatherReader::ReadMetadata() const {
- const int num_metadata_fields = 4;
- const char* fieldnames[] = {"NumRows", "NumVariables", "Description", "Version"};
+ const int32_t num_metadata_fields = 3;
+ const char* fieldnames[] = {"NumRows", "NumVariables", "Description"};
// Create a mxArray struct array containing the table metadata to be passed back to
// MATLAB.
@@ -176,37 +255,36 @@ mxArray* FeatherReader::ReadMetadata() const {
mxCreateDoubleScalar(static_cast<double>(num_variables_)));
// Set the description.
- mxSetField(metadata, 0, "Description", mxCreateString(description_.c_str()));
-
- // Set the version.
- mxSetField(metadata, 0, "Version", mxCreateDoubleScalar(static_cast<double>(version_)));
+ mxSetField(metadata, 0, "Description",
+ util::ConvertUTF8StringToUTF16CharMatrix(description_));
return metadata;
}
// Read the table variables from the Feather file as a mxArray*.
mxArray* FeatherReader::ReadVariables() const {
- const int num_variable_fields = 4;
- const char* fieldnames[] = {"Name", "Data", "Nulls", "Type"};
+ const int32_t num_variable_fields = 4;
+ const char* fieldnames[] = {"Name", "Type", "Data", "Valid"};
- // Create an mxArray struct array containing the table variables to be passed back to
+ // Create an mxArray* struct array containing the table variables to be passed back to
// MATLAB.
mxArray* variables =
mxCreateStructMatrix(1, num_variables_, num_variable_fields, fieldnames);
// Read all the table variables in the Feather file into memory.
for (int64_t i = 0; i < num_variables_; ++i) {
- std::shared_ptr<arrow::Column> column(nullptr);
+ std::shared_ptr<Column> column(nullptr);
util::HandleStatus(table_reader_->GetColumn(i, &column));
// set the struct fields data
mxSetField(variables, i, "Name", internal::ReadVariableName(column));
- mxSetField(variables, i, "Data", internal::ReadVariableData(column));
- mxSetField(variables, i, "Nulls", internal::ReadVariableNulls(column));
mxSetField(variables, i, "Type", internal::ReadVariableType(column));
+ mxSetField(variables, i, "Data", internal::ReadVariableData(column));
+ mxSetField(variables, i, "Valid", internal::ReadVariableValidityBitmap(column));
}
return variables;
}
-} // namespace mlarrow
+} // namespace matlab
+} // namespace arrow
diff --git a/matlab/src/feather_reader.h b/matlab/src/feather_reader.h
index 51405b5..c676142 100644
--- a/matlab/src/feather_reader.h
+++ b/matlab/src/feather_reader.h
@@ -15,8 +15,8 @@
// specific language governing permissions and limitations
// under the License.
-#ifndef MLARROW_FEATHER_READER_H
-#define MLARROW_FEATHER_READER_H
+#ifndef ARROW_MATLAB_FEATHER_READER_H
+#define ARROW_MATLAB_FEATHER_READER_H
#include <memory>
#include <string>
@@ -27,45 +27,53 @@
#include <matrix.h>
-namespace mlarrow {
+namespace arrow {
+namespace matlab {
class FeatherReader {
public:
~FeatherReader() = default;
- /// \brief Read the table metadata as a mxArray* struct from the given Feather file.
- /// The returned struct includes fields describing the number of rows
- /// in the table (NumRows), the number of variables (NumVariables), the
- /// Feather file version (Version), and the table description (Description).
- /// Callers are responsible for freeing the returned mxArray memory
+ /// \brief Read the table metadata as an mxArray* struct from the given
+ /// Feather file.
+ /// The returned mxArray* struct contains the following fields:
+ /// - "Description" :: Nx1 mxChar array, table-level description
+ /// - "NumRows" :: scalar mxDouble array, number of rows in the
+ /// table
+ /// - "NumVariables" :: scalar mxDouble array, number of variables in
+ /// the table
+ /// Clients are responsible for freeing the returned mxArray memory
/// when it is no longer needed, or passing it to MATLAB to be managed.
/// \return metadata mxArray* scalar struct containing table level metadata
mxArray* ReadMetadata() const;
- /// \brief Read the table metadata as a mxArray* struct array from the given
- /// Feather file. Each struct includes fields for variable data (Data),
- /// null values (Nulls), variable name (Name), and original Feather
- /// datatype (Type). Callers are responsible for freeing the returned
- /// mxArray memory when it is no longer needed, or passing it to MATLAB
- /// to be managed.
+ /// \brief Read the table variable data as an mxArray* struct array from the
+ /// given Feather file.
+ /// The returned mxArray* struct array has the following fields:
+ /// - "Name" :: Nx1 mxChar array, name of the variable
+ /// - "Type" :: Nx1 mxChar array, the variable's Arrow datatype
+ /// - "Data" :: Nx1 mxArray, data for the variable
+ /// - "Valid" :: Nx1 mxLogical array, validity (null) bitmap
+ /// Clients are responsible for freeing the returned mxArray memory
+ /// when it is no longer needed, or passing it to MATLAB to be managed.
/// \return variables mxArray* struct array containing table variable data
mxArray* ReadVariables() const;
/// \brief Initialize a FeatherReader object from a given Feather file.
/// \param[in] filename path to a Feather file
/// \param[out] feather_reader uninitialized FeatherReader object
- static arrow::Status Open(const std::string& filename,
- std::shared_ptr<FeatherReader>* feather_reader);
+ static Status Open(const std::string& filename,
+ std::shared_ptr<FeatherReader>* feather_reader);
private:
FeatherReader() = default;
- std::unique_ptr<arrow::ipc::feather::TableReader> table_reader_;
+ std::unique_ptr<ipc::feather::TableReader> table_reader_;
int64_t num_rows_;
int64_t num_variables_;
std::string description_;
- int version_;
};
-} // namespace mlarrow
+} // namespace matlab
+} // namespace arrow
-#endif // MLARROW_FEATHER_READER_H
+#endif // ARROW_MATLAB_FEATHER_READER_H
diff --git a/matlab/src/feather_writer.cc b/matlab/src/feather_writer.cc
new file mode 100644
index 0000000..215bc6d
--- /dev/null
+++ b/matlab/src/feather_writer.cc
@@ -0,0 +1,338 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <cmath>
+#include <functional> /* for std::multiplies */
+#include <numeric> /* for std::accumulate */
+
+#include <arrow/array.h>
+#include <arrow/buffer.h>
+#include <arrow/io/file.h>
+#include <arrow/ipc/feather.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/type.h>
+#include <arrow/util/bit-util.h>
+
+#include <mex.h>
+
+#include "feather_writer.h"
+#include "matlab_traits.h"
+#include "util/handle_status.h"
+
+namespace arrow {
+namespace matlab {
+namespace internal {
+
+// Utility that helps verify the input mxArray struct field name and type.
+// Returns void since any errors will throw and terminate MEX execution.
+void ValidateMxStructField(const mxArray* struct_array, const char* fieldname,
+ mxClassID expected_class_id, bool can_be_empty) {
+ // Check that the input mxArray is a struct array.
+ if (!mxIsStruct(struct_array)) {
+ mexErrMsgIdAndTxt("MATLAB:arrow:IncorrectDimensionsOrType",
+ "Input needs to be a struct array");
+ }
+
+ // Return early if an empty table is provided as input.
+ if (mxIsEmpty(struct_array)) {
+ return;
+ }
+
+ mxArray* field = mxGetField(struct_array, 0, fieldname);
+
+ if (!field) {
+ mexErrMsgIdAndTxt("MATLAB:arrow:MissingStructField",
+ "Missing field '%s' in input struct array", fieldname);
+ }
+
+ mxClassID actual_class_id = mxGetClassID(field);
+
+ // Avoid type check if an mxUNKNOWN_CLASS is provided since the UNKNOWN type is used to
+ // signify genericity in the input type.
+ if (expected_class_id != mxUNKNOWN_CLASS) {
+ if (expected_class_id != actual_class_id) {
+ mexErrMsgIdAndTxt("MATLAB:arrow:MissingStructField",
+ "Incorrect type '%s' for struct array field '%s'",
+ mxGetClassName(field), fieldname);
+ }
+ }
+
+ // Some struct fields (like the table description) can be empty, while others
+ // (like NumRows) should never be empty. This conditional helps account for both cases.
+ if (!can_be_empty) {
+ // Ensure that individual mxStructArray fields are non-empty.
+ // We can call mxGetData after this without needing another null check.
+ if (mxIsEmpty(field)) {
+ mexErrMsgIdAndTxt("MATLAB:arrow:EmptyStructField",
+ "Struct array field '%s' cannot be empty", fieldname);
+ }
+ }
+}
+
+// Utility function to convert mxChar mxArray* to std::string while preserving
+// Unicode code points.
+std::string MxArrayToString(const mxArray* array) {
+ // Return empty std::string if a mxChar array is not passed in.
+ if (!mxIsChar(array)) {
+ return std::string();
+ }
+
+ // Convert mxArray first to a C-style char array, then copy into a std::string.
+ char* utf8_array = mxArrayToUTF8String(array);
+ std::string output(utf8_array);
+
+ // Free the allocated char* from the MEX runtime.
+ mxFree(utf8_array);
+
+ return output;
+}
+
+// Compare number of columns and exit out to the MATLAB layer if incorrect.
+void ValidateNumColumns(int64_t actual, int64_t expected) {
+ if (actual != expected) {
+ mexErrMsgIdAndTxt("MATLAB:arrow:IncorrectNumberOfColumns",
+ "Received only '%d' columns but expected '%d' columns", actual,
+ expected);
+ }
+}
+
+// Compare number of rows and exit out to the MATLAB layer if incorrect.
+void ValidateNumRows(int64_t actual, int64_t expected) {
+ if (actual != expected) {
+ mexErrMsgIdAndTxt("MATLAB:arrow:IncorrectNumberOfRows",
+ "Received only '%d' rows but expected '%d' rows", actual, expected);
+ }
+}
+
+// Calculate the number of bytes required in the bit-packed validity buffer.
+constexpr int64_t BitPackedLength(int64_t num_elements) {
+ // Since mxLogicalArray encodes [0, 1] in a full byte, we can compress that byte
+ // down to a bit...therefore dividing the mxLogicalArray length by 8 here.
+ return static_cast<int64_t>(std::ceil(num_elements / 8.0));
+}
+
+// Calculate the total number of elements in an mxArray
+// We have to do this separately since mxGetNumberOfElements only works in numeric arrays
+size_t GetNumberOfElements(const mxArray* array) {
+ // Get the dimensions and the total number of dimensions from the mxArray*.
+ const size_t num_dimensions = mxGetNumberOfDimensions(array);
+ const size_t* dimensions = mxGetDimensions(array);
+
+ // Iterate over the dimensions array and accumulate the total number of elements.
+ return std::accumulate(dimensions, dimensions + num_dimensions, 1,
+ std::multiplies<size_t>());
+}
+
+// Write an mxLogicalArray* into a bit-packed arrow::MutableBuffer
+void BitPackBuffer(const mxArray* logical_array,
+ std::shared_ptr<MutableBuffer> packed_buffer) {
+ // Error out if the incorrect type is passed in.
+ if (!mxIsLogical(logical_array)) {
+ mexErrMsgIdAndTxt(
+ "MATLAB:arrow:IncorrectType",
+ "Expected mxLogical array as input but received mxArray of class '%s'",
+ mxGetClassName(logical_array));
+ }
+
+ // Validate that the input arrow::Buffer has sufficent size to store a full bit-packed
+ // representation of the input mxLogicalArray
+ int64_t unpacked_buffer_length = GetNumberOfElements(logical_array);
+ if (BitPackedLength(unpacked_buffer_length) > packed_buffer->capacity()) {
+ mexErrMsgIdAndTxt("MATLAB:arrow:BufferSizeExceeded",
+ "Buffer of size %d bytes cannot store %d bytes of data",
+ packed_buffer->capacity(), BitPackedLength(unpacked_buffer_length));
+ }
+
+ // Get pointers to the internal uint8_t arrays behind arrow::Buffer and mxArray
+ uint8_t* packed_buffer_ptr = packed_buffer->mutable_data();
+ mxLogical* unpacked_buffer_ptr = mxGetLogicals(logical_array);
+
+ // Iterate over the mxLogical array and write bit-packed bools to the arrow::Buffer.
+ // Call into a loop-unrolled Arrow utility for better performance when bit-packing.
+ auto generator = [&]() -> uint8_t { return *unpacked_buffer_ptr++; };
+ const int64_t start_offset = 0;
+ arrow::internal::GenerateBitsUnrolled(packed_buffer_ptr, start_offset,
+ unpacked_buffer_length, generator);
+}
+
+// Write numeric datatypes to the Feather file.
+template <typename ArrowDataType>
+std::unique_ptr<Array> WriteNumericData(const mxArray* data,
+ const std::shared_ptr<Buffer> validity_bitmap) {
+ // Alias the type name for the underlying MATLAB type.
+ using MatlabType = typename MatlabTraits<ArrowDataType>::MatlabType;
+
+ // Get a pointer to the underlying mxArray data.
+ // We need to (temporarily) cast away const here since the mxGet* functions do not
+ // accept a const input parameter for compatibility reasons.
+ const MatlabType* dt = MatlabTraits<ArrowDataType>::GetData(const_cast<mxArray*>(data));
+
+ // Construct an arrow::Buffer that points to the underlying mxArray without copying.
+ // - The lifetime of the mxArray buffer exceeds that of the arrow::Buffer here since
+ // MATLAB should only free this region on garbage-collection after the MEX function
+ // is executed. Therefore it is safe for arrow::Buffer to point to this location.
+ // - However arrow::Buffer must not free this region by itself, since that could cause
+ // segfaults if the input array is used later in MATLAB.
+ // - The Doxygen doc for arrow::Buffer's constructor implies that it is not an RAII
+ // type, so this should be safe from possible double-free here.
+ std::shared_ptr<Buffer> buffer =
+ std::make_shared<Buffer>(reinterpret_cast<const uint8_t*>(dt),
+ mxGetElementSize(data) * mxGetNumberOfElements(data));
+
+ // Construct arrow::NumericArray specialization using arrow::Buffer.
+ // Pass in nulls information...we could compute and provide the number of nulls here too,
+ // but passing -1 for now so that Arrow recomputes it if necessary.
+ return std::unique_ptr<Array>(new NumericArray<ArrowDataType>(
+ mxGetNumberOfElements(data), buffer, validity_bitmap, -1));
+}
+
+// Dispatch MATLAB column data to the correct arrow::Array converter.
+std::unique_ptr<Array> WriteVariableData(const mxArray* data, const std::string& type,
+ const std::shared_ptr<Buffer> validity_bitmap) {
+ // Get the underlying type of the mxArray data.
+ const mxClassID mxclass = mxGetClassID(data);
+
+ switch (mxclass) {
+ case mxSINGLE_CLASS:
+ return WriteNumericData<FloatType>(data, validity_bitmap);
+ case mxDOUBLE_CLASS:
+ return WriteNumericData<DoubleType>(data, validity_bitmap);
+ case mxUINT8_CLASS:
+ return WriteNumericData<UInt8Type>(data, validity_bitmap);
+ case mxUINT16_CLASS:
+ return WriteNumericData<UInt16Type>(data, validity_bitmap);
+ case mxUINT32_CLASS:
+ return WriteNumericData<UInt32Type>(data, validity_bitmap);
+ case mxUINT64_CLASS:
+ return WriteNumericData<UInt64Type>(data, validity_bitmap);
+ case mxINT8_CLASS:
+ return WriteNumericData<Int8Type>(data, validity_bitmap);
+ case mxINT16_CLASS:
+ return WriteNumericData<Int16Type>(data, validity_bitmap);
+ case mxINT32_CLASS:
+ return WriteNumericData<Int32Type>(data, validity_bitmap);
+ case mxINT64_CLASS:
+ return WriteNumericData<Int64Type>(data, validity_bitmap);
+
+ default: {
+ mexErrMsgIdAndTxt("MATLAB:arrow:UnsupportedArrowType",
+ "Unsupported arrow::Type '%s' for variable '%s'",
+ mxGetClassName(data), type.c_str());
+ }
+ }
+
+ // We shouldn't ever reach this branch, but if we do, return nullptr.
+ return nullptr;
+}
+
+} // namespace internal
+
+Status FeatherWriter::Open(const std::string& filename,
+ std::shared_ptr<FeatherWriter>* feather_writer) {
+ // Allocate shared_ptr out parameter.
+ *feather_writer = std::shared_ptr<FeatherWriter>(new FeatherWriter());
+
+ // Open a FileOutputStream corresponding to the provided filename.
+ std::shared_ptr<io::OutputStream> writable_file(nullptr);
+ ARROW_RETURN_NOT_OK(io::FileOutputStream::Open(filename, &writable_file));
+
+ // TableWriter::Open expects a shared_ptr to an OutputStream.
+ // Open the Feather file for writing with a TableWriter.
+ return ipc::feather::TableWriter::Open(writable_file,
+ &(*feather_writer)->table_writer_);
+}
+
+// Write table metadata to the Feather file from a mxArray*.
+void FeatherWriter::WriteMetadata(const mxArray* metadata) {
+ // Verify that all required fieldnames are provided.
+ internal::ValidateMxStructField(metadata, "Description", mxCHAR_CLASS, true);
+ internal::ValidateMxStructField(metadata, "NumRows", mxDOUBLE_CLASS, false);
+ internal::ValidateMxStructField(metadata, "NumVariables", mxDOUBLE_CLASS, false);
+
+ // Convert Description to a std::string and set on FeatherWriter and TableWriter.
+ std::string description =
+ internal::MxArrayToString(mxGetField(metadata, 0, "Description"));
+ this->description_ = description;
+ this->table_writer_->SetDescription(description);
+
+ // Get the NumRows field in the struct array and set on TableWriter.
+ this->num_rows_ = static_cast<int64_t>(mxGetScalar(mxGetField(metadata, 0, "NumRows")));
+ this->table_writer_->SetNumRows(this->num_rows_);
+
+ // Get the total number of variables. This is checked later for consistency with
+ // the provided number of columns before finishing the file write.
+ this->num_variables_ =
+ static_cast<int64_t>(mxGetScalar(mxGetField(metadata, 0, "NumVariables")));
+}
+
+// Write mxArrays from MATLAB into a Feather file.
+Status FeatherWriter::WriteVariables(const mxArray* variables) {
+ // Verify that all required fieldnames are provided.
+ internal::ValidateMxStructField(variables, "Name", mxCHAR_CLASS, true);
+ internal::ValidateMxStructField(variables, "Type", mxCHAR_CLASS, false);
+ internal::ValidateMxStructField(variables, "Data", mxUNKNOWN_CLASS, true);
+ internal::ValidateMxStructField(variables, "Valid", mxLOGICAL_CLASS, true);
+
+ // Get the number of columns in the struct array.
+ size_t num_columns = internal::GetNumberOfElements(variables);
+
+ // Verify that we have all the columns required for writing
+ // Currently we need all columns to be passed in together in the WriteVariables method.
+ internal::ValidateNumColumns(static_cast<int64_t>(num_columns), this->num_variables_);
+
+ // Allocate a packed validity bitmap for later arrow::Buffers to reference and populate.
+ // Since this is defined in the enclosing scope around any arrow::Buffer usage, this
+ // should outlive any arrow::Buffers created on this range, thus avoiding dangling
+ // references.
+ std::shared_ptr<ResizableBuffer> validity_bitmap;
+ ARROW_RETURN_NOT_OK(AllocateResizableBuffer(internal::BitPackedLength(this->num_rows_),
+ &validity_bitmap));
+
+ // Iterate over the input columns and generate arrow arrays.
+ for (int idx = 0; idx < num_columns; ++idx) {
+ // Unwrap constituent mxArray*s from the mxStructArray*. This is safe since we
+ // already checked for existence and non-nullness of these types.
+ const mxArray* name = mxGetField(variables, idx, "Name");
+ const mxArray* data = mxGetField(variables, idx, "Data");
+ const mxArray* type = mxGetField(variables, idx, "Type");
+ const mxArray* valid = mxGetField(variables, idx, "Valid");
+
+ // Convert column and type name to a std::string from mxArray*.
+ std::string name_str = internal::MxArrayToString(name);
+ std::string type_str = internal::MxArrayToString(type);
+
+ // Populate bit-packed arrow::Buffer using validity data in the mxArray*.
+ internal::BitPackBuffer(valid, validity_bitmap);
+
+ // Wrap mxArray data in an arrow::Array of the equivalent type.
+ std::unique_ptr<Array> array =
+ internal::WriteVariableData(data, type_str, validity_bitmap);
+
+ // Verify that the arrow::Array has the right number of elements.
+ internal::ValidateNumRows(array->length(), this->num_rows_);
+
+ // Write another column to the Feather file.
+ ARROW_RETURN_NOT_OK(this->table_writer_->Append(name_str, *array));
+ }
+
+ // Write the Feather file metadata to the end of the file.
+ return this->table_writer_->Finalize();
+}
+
+} // namespace matlab
+} // namespace arrow
diff --git a/matlab/src/feather_writer.h b/matlab/src/feather_writer.h
new file mode 100644
index 0000000..5d25712
--- /dev/null
+++ b/matlab/src/feather_writer.h
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef ARROW_MATLAB_FEATHER_WRITER_H
+#define ARROW_MATLAB_FEATHER_WRITER_H
+
+#include <memory>
+#include <string>
+
+#include <arrow/ipc/feather.h>
+#include <arrow/status.h>
+#include <arrow/type.h>
+
+#include <matrix.h>
+
+namespace arrow {
+namespace matlab {
+
+class FeatherWriter {
+ public:
+ ~FeatherWriter() = default;
+
+ /// \brief Write Feather file metadata using information from an mxArray* struct.
+ /// The input mxArray must be a scalar struct array with the following fields:
+ /// - "Description" :: Nx1 mxChar array, table-level description
+ /// - "NumRows" :: scalar mxDouble array, number of rows in table
+ /// - "NumVariables" :: scalar mxDouble array, total number of variables
+ /// \param[in] metadata mxArray* scalar struct containing table-level metadata
+ void WriteMetadata(const mxArray* metadata);
+
+ /// \brief Write mxArrays to a Feather file. The input must be a N-by-1 mxStruct
+ // array with the following fields:
+ /// - "Name" :: Nx1 mxChar array, name of the column
+ /// - "Type" :: Nx1 mxChar array, the variable's MATLAB datatype
+ /// - "Data" :: Nx1 mxArray, data for this variable
+ /// - "Valid" :: Nx1 mxLogical array, 0 represents invalid (null) values and
+ /// 1 represents valid (non-null) values
+ /// \param[in] variables mxArray* struct array containing table variable data
+ /// \return status
+ Status WriteVariables(const mxArray* variables);
+
+ /// \brief Initialize a FeatherWriter object that writes to a Feather file
+ /// \param[in] filename path to the new Feather file
+ /// \param[out] feather_writer uninitialized FeatherWriter object
+ /// \return status
+ static Status Open(const std::string& filename,
+ std::shared_ptr<FeatherWriter>* feather_writer);
+
+ private:
+ FeatherWriter() = default;
+
+ std::unique_ptr<ipc::feather::TableWriter> table_writer_;
+ int64_t num_rows_;
+ int64_t num_variables_;
+ std::string description_;
+};
+
+} // namespace matlab
+} // namespace arrow
+
+#endif // ARROW_MATLAB_FEATHER_WRITER_H
diff --git a/matlab/src/featherread.m b/matlab/src/featherread.m
index ababdcc..4ac8a56 100644
--- a/matlab/src/featherread.m
+++ b/matlab/src/featherread.m
@@ -23,6 +23,8 @@ function t = featherread(filename)
% specific language governing permissions and limitations
% under the License.
+import mlarrow.util.*;
+
% Validate input arguments.
narginchk(1, 1);
filename = convertStringsToChars(filename);
@@ -46,14 +48,19 @@ end
% libarrow.
[variables, metadata] = featherreadmex(filename);
-% Preallocate a cell array for the case in which
-% the table VariableDescriptions property needs to be modified.
-variableDescriptions = cell(1, numel(variables));
+% Make valid MATLAB table variable names out of any of the
+% Feather table column names that are not valid MATLAB table
+% variable names.
+[variableNames, variableDescriptions] = makeValidMATLABTableVariableNames({variables.Name});
-% Iterate over each table variable, handling null entries and invalid
-% variable names appropriately.
+% Iterate over each table variable, handling invalid (null) entries
+% and invalid MATLAB table variable names appropriately.
+% Note: All Arrow arrays can have an associated validity (null) bitmap.
+% The Apache Arrow specification defines 0 (false) to represent an
+% invalid (null) array entry and 1 (true) to represent a valid
+% (non-null) array entry.
for ii = 1:length(variables)
- if any(variables(ii).Nulls)
+ if ~all(variables(ii).Valid)
switch variables(ii).Type
case {'uint8', 'uint16', 'uint32', 'uint64', 'int8', 'int16', 'int32', 'int64'}
% MATLAB does not support missing values for integer types, so
@@ -61,31 +68,21 @@ for ii = 1:length(variables)
variables(ii).Data = double(variables(ii).Data);
end
- % Set null entries to the appropriate MATLAB missing value using
+ % Set invalid (null) entries to the appropriate MATLAB missing value using
% logical indexing.
- variables(ii).Data(variables(ii).Nulls) = missing;
- end
-
- % Store invalid variable names in the VariableDescriptons
- % property, and convert any invalid variable names into valid variable
- % names.
- setVariableDescriptions = false;
- if ~isvarname(variables(ii).Name)
- variableDescriptions{ii} = sprintf('Original variable name: ''%s''', variables(ii).Name);
- setVariableDescriptions = true;
- else
- variableDescriptions{ii} = '';
+ variables(ii).Data(~variables(ii).Valid) = missing;
end
end
% Construct a MATLAB table from the Feather file data.
-t = table(variables.Data, 'VariableNames', matlab.lang.makeValidName({variables.Name}));
+t = table(variables.Data, 'VariableNames', cellstr(variableNames));
-% Store original variable names in the VariableDescriptions property
-% if they were modified to be valid MATLAB table variable names.
-if setVariableDescriptions
- t.Properties.VariableDescriptions = variableDescriptions;
+% Store original Feather table column names in the table.Properties.VariableDescriptions
+% property if they were modified to be valid MATLAB table variable names.
+if ~isempty(variableDescriptions)
+ t.Properties.VariableDescriptions = cellstr(variableDescriptions);
end
+
% Set the Description property of the table based on the Feather file
% description.
t.Properties.Description = metadata.Description;
diff --git a/matlab/src/featherreadmex.cc b/matlab/src/featherreadmex.cc
index 0499a37..b52b8a9 100644
--- a/matlab/src/featherreadmex.cc
+++ b/matlab/src/featherreadmex.cc
@@ -27,8 +27,9 @@ void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const mxArray* prhs[]) {
const std::string filename{mxArrayToUTF8String(prhs[0])};
// Read the given Feather file into memory.
- std::shared_ptr<mlarrow::FeatherReader> feather_reader{nullptr};
- mlarrow::util::HandleStatus(mlarrow::FeatherReader::Open(filename, &feather_reader));
+ std::shared_ptr<arrow::matlab::FeatherReader> feather_reader{nullptr};
+ arrow::matlab::util::HandleStatus(
+ arrow::matlab::FeatherReader::Open(filename, &feather_reader));
// Return the Feather file table variables and table metadata to MATLAB.
plhs[0] = feather_reader->ReadVariables();
diff --git a/matlab/src/featherwrite.m b/matlab/src/featherwrite.m
new file mode 100644
index 0000000..eeedf26
--- /dev/null
+++ b/matlab/src/featherwrite.m
@@ -0,0 +1,44 @@
+function featherwrite(filename, t)
+%FEATHERWRITE Write a table to a Feather file.
+% Use the FEATHERWRITE function to write a table to
+% a Feather file as column-oriented data.
+%
+% FEATHERWRITE(FILENAME,T) writes the table T to a Feather
+% file FILENAME as column-oriented data.
+
+% Licensed to the Apache Software Foundation (ASF) under one
+% or more contributor license agreements. See the NOTICE file
+% distributed with this work for additional information
+% regarding copyright ownership. The ASF licenses this file
+% to you under the Apache License, Version 2.0 (the
+% "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing,
+% software distributed under the License is distributed on an
+% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+% KIND, either express or implied. See the License for the
+% specific language governing permissions and limitations
+% under the License.
+
+import mlarrow.util.table2mlarrow;
+
+% Validate input arguments.
+narginchk(2, 2);
+filename = convertStringsToChars(filename);
+if ~ischar(filename)
+ error('MATLAB:arrow:InvalidFilenameDatatype', ...
+ 'Filename must be a character vector or string scalar.');
+end
+if ~istable(t)
+ error('MATLAB:arrow:InvalidInputTable', 't must be a table.');
+end
+
+[variables, metadata] = table2mlarrow(t);
+
+% Write the table to a Feather file.
+featherwritemex(filename, variables, metadata);
+
+end
diff --git a/matlab/src/featherreadmex.cc b/matlab/src/featherwritemex.cc
similarity index 65%
copy from matlab/src/featherreadmex.cc
copy to matlab/src/featherwritemex.cc
index 0499a37..3a6815e 100644
--- a/matlab/src/featherreadmex.cc
+++ b/matlab/src/featherwritemex.cc
@@ -19,18 +19,19 @@
#include <mex.h>
-#include "feather_reader.h"
+#include "feather_writer.h"
#include "util/handle_status.h"
-// MEX gateway function. This is the entry point for featherreadmex.cpp.
+// MEX gateway function. This is the entry point for featherwritemex.cc.
void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const mxArray* prhs[]) {
const std::string filename{mxArrayToUTF8String(prhs[0])};
- // Read the given Feather file into memory.
- std::shared_ptr<mlarrow::FeatherReader> feather_reader{nullptr};
- mlarrow::util::HandleStatus(mlarrow::FeatherReader::Open(filename, &feather_reader));
+ // Open a Feather file at the provided file path for writing.
+ std::shared_ptr<arrow::matlab::FeatherWriter> feather_writer{nullptr};
+ arrow::matlab::util::HandleStatus(
+ arrow::matlab::FeatherWriter::Open(filename, &feather_writer));
- // Return the Feather file table variables and table metadata to MATLAB.
- plhs[0] = feather_reader->ReadVariables();
- plhs[1] = feather_reader->ReadMetadata();
+ // Write the Feather file table variables and table metadata from MATLAB.
+ feather_writer->WriteMetadata(prhs[2]);
+ arrow::matlab::util::HandleStatus(feather_writer->WriteVariables(prhs[1]));
}
diff --git a/matlab/src/matlab_traits.h b/matlab/src/matlab_traits.h
index 7d5987a..6cdfacc 100644
--- a/matlab/src/matlab_traits.h
+++ b/matlab/src/matlab_traits.h
@@ -15,89 +15,91 @@
// specific language governing permissions and limitations
// under the License.
-#ifndef MLARROW_MATLAB_TRAITS_H
-#define MLARROW_MATLAB_TRAITS_H
+#ifndef ARROW_MATLAB_MATLAB_TRAITS_H
+#define ARROW_MATLAB_MATLAB_TRAITS_H
#include <arrow/type.h>
#include <matrix.h>
-namespace mlarrow {
+namespace arrow {
+namespace matlab {
/// \brief A type traits class mapping Arrow types to MATLAB types.
template <typename ArrowDataType>
struct MatlabTraits;
template <>
-struct MatlabTraits<arrow::FloatType> {
+struct MatlabTraits<FloatType> {
static constexpr mxClassID matlab_class_id = mxSINGLE_CLASS;
- typedef mxSingle MatlabType;
+ using MatlabType = mxSingle;
static MatlabType* GetData(mxArray* pa) { return mxGetSingles(pa); }
};
template <>
-struct MatlabTraits<arrow::DoubleType> {
+struct MatlabTraits<DoubleType> {
static constexpr mxClassID matlab_class_id = mxDOUBLE_CLASS;
- typedef mxDouble MatlabType;
+ using MatlabType = mxDouble;
static MatlabType* GetData(mxArray* pa) { return mxGetDoubles(pa); }
};
template <>
-struct MatlabTraits<arrow::UInt8Type> {
+struct MatlabTraits<UInt8Type> {
static constexpr mxClassID matlab_class_id = mxUINT8_CLASS;
- typedef mxUint8 MatlabType;
+ using MatlabType = mxUint8;
static MatlabType* GetData(mxArray* pa) { return mxGetUint8s(pa); }
};
template <>
-struct MatlabTraits<arrow::UInt16Type> {
+struct MatlabTraits<UInt16Type> {
static constexpr mxClassID matlab_class_id = mxUINT16_CLASS;
- typedef mxUint16 MatlabType;
+ using MatlabType = mxUint16;
static MatlabType* GetData(mxArray* pa) { return mxGetUint16s(pa); }
};
template <>
-struct MatlabTraits<arrow::UInt32Type> {
+struct MatlabTraits<UInt32Type> {
static constexpr mxClassID matlab_class_id = mxUINT32_CLASS;
- typedef mxUint32 MatlabType;
+ using MatlabType = mxUint32;
static MatlabType* GetData(mxArray* pa) { return mxGetUint32s(pa); }
};
template <>
-struct MatlabTraits<arrow::UInt64Type> {
+struct MatlabTraits<UInt64Type> {
static constexpr mxClassID matlab_class_id = mxUINT64_CLASS;
- typedef mxUint64 MatlabType;
+ using MatlabType = mxUint64;
static MatlabType* GetData(mxArray* pa) { return mxGetUint64s(pa); }
};
template <>
-struct MatlabTraits<arrow::Int8Type> {
+struct MatlabTraits<Int8Type> {
static constexpr mxClassID matlab_class_id = mxINT8_CLASS;
- typedef mxInt8 MatlabType;
+ using MatlabType = mxInt8;
static MatlabType* GetData(mxArray* pa) { return mxGetInt8s(pa); }
};
template <>
-struct MatlabTraits<arrow::Int16Type> {
+struct MatlabTraits<Int16Type> {
static constexpr mxClassID matlab_class_id = mxINT16_CLASS;
- typedef mxInt16 MatlabType;
+ using MatlabType = mxInt16;
static MatlabType* GetData(mxArray* pa) { return mxGetInt16s(pa); }
};
template <>
-struct MatlabTraits<arrow::Int32Type> {
+struct MatlabTraits<Int32Type> {
static constexpr mxClassID matlab_class_id = mxINT32_CLASS;
- typedef mxInt32 MatlabType;
+ using MatlabType = mxInt32;
static MatlabType* GetData(mxArray* pa) { return mxGetInt32s(pa); }
};
template <>
-struct MatlabTraits<arrow::Int64Type> {
+struct MatlabTraits<Int64Type> {
static constexpr mxClassID matlab_class_id = mxINT64_CLASS;
- typedef mxInt64 MatlabType;
+ using MatlabType = mxInt64;
static MatlabType* GetData(mxArray* pa) { return mxGetInt64s(pa); }
};
-} // namespace mlarrow
+} // namespace matlab
+} // namespace arrow
-#endif // MLARROW_MATLAB_TRAITS_H
+#endif // ARROW_MATLAB_MATLAB_TRAITS_H
diff --git a/matlab/src/util/handle_status.cc b/matlab/src/util/handle_status.cc
index 6c11b4d..992f2c3 100644
--- a/matlab/src/util/handle_status.cc
+++ b/matlab/src/util/handle_status.cc
@@ -19,87 +19,87 @@
#include <mex.h>
-namespace mlarrow {
-
+namespace arrow {
+namespace matlab {
namespace util {
-void HandleStatus(const arrow::Status& status) {
+void HandleStatus(const Status& status) {
const char* arrow_error_message = "Arrow error: %s";
switch (status.code()) {
- case arrow::StatusCode::OK: {
+ case StatusCode::OK: {
break;
}
- case arrow::StatusCode::OutOfMemory: {
+ case StatusCode::OutOfMemory: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:OutOfMemory", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::KeyError: {
+ case StatusCode::KeyError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:KeyError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::TypeError: {
+ case StatusCode::TypeError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:TypeError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::Invalid: {
+ case StatusCode::Invalid: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:Invalid", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::IOError: {
+ case StatusCode::IOError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:IOError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::CapacityError: {
+ case StatusCode::CapacityError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:CapacityError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::IndexError: {
+ case StatusCode::IndexError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:IndexError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::UnknownError: {
+ case StatusCode::UnknownError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:UnknownError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::NotImplemented: {
+ case StatusCode::NotImplemented: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:NotImplemented", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::SerializationError: {
+ case StatusCode::SerializationError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:SerializationError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::PythonError: {
+ case StatusCode::PythonError: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:PythonError", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::PlasmaObjectExists: {
+ case StatusCode::PlasmaObjectExists: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:PlasmaObjectExists", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::PlasmaObjectNonexistent: {
+ case StatusCode::PlasmaObjectNonexistent: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:PlasmaObjectNonexistent",
arrow_error_message, status.ToString().c_str());
break;
}
- case arrow::StatusCode::PlasmaStoreFull: {
+ case StatusCode::PlasmaStoreFull: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:PlasmaStoreFull", arrow_error_message,
status.ToString().c_str());
break;
}
- case arrow::StatusCode::PlasmaObjectAlreadySealed: {
+ case StatusCode::PlasmaObjectAlreadySealed: {
mexErrMsgIdAndTxt("MATLAB:arrow:status:PlasmaObjectAlreadySealed",
arrow_error_message, status.ToString().c_str());
break;
@@ -112,4 +112,5 @@ void HandleStatus(const arrow::Status& status) {
}
}
} // namespace util
-} // namespace mlarrow
+} // namespace matlab
+} // namespace arrow
diff --git a/matlab/src/util/handle_status.h b/matlab/src/util/handle_status.h
index 18b0773..7485803 100644
--- a/matlab/src/util/handle_status.h
+++ b/matlab/src/util/handle_status.h
@@ -15,18 +15,20 @@
// specific language governing permissions and limitations
// under the License.
-#ifndef MLARROW_UTIL_HANDLE_STATUS_H
-#define MLARROW_UTIL_HANDLE_STATUS_H
+#ifndef ARROW_MATLAB_UTIL_HANDLE_STATUS_H
+#define ARROW_MATLAB_UTIL_HANDLE_STATUS_H
#include <arrow/status.h>
-namespace mlarrow {
+namespace arrow {
+namespace matlab {
namespace util {
// Terminates execution and returns to the MATLAB prompt,
// displaying an error message if the given status
// indicates that an error has occurred.
-void HandleStatus(const arrow::Status& status);
+void HandleStatus(const Status& status);
} // namespace util
-} // namespace mlarrow
+} // namespace matlab
+} // namespace arrow
-#endif // MLARROW_UTIL_HANDLE_STATUS_H
+#endif // ARROW_MATLAB_UTIL_HANDLE_STATUS_H
diff --git a/matlab/src/util/unicode_conversion.cc b/matlab/src/util/unicode_conversion.cc
new file mode 100644
index 0000000..01c2e4b
--- /dev/null
+++ b/matlab/src/util/unicode_conversion.cc
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <locale> /* for std::wstring_convert */
+#include <codecvt> /* for std::codecvt_utf8_utf16 */
+
+#include "unicode_conversion.h"
+
+namespace arrow {
+namespace matlab {
+namespace util {
+
+mxArray* ConvertUTF8StringToUTF16CharMatrix(const std::string& utf8_string) {
+ // Get pointers to the start and end of the std::string data.
+ const char* string_start = utf8_string.c_str();
+ const char* string_end = string_start + utf8_string.length();
+
+ // Due to this issue on MSVC: https://stackoverflow.com/q/32055357 we cannot
+ // directly use a destination type of char16_t.
+#if _MSC_VER >= 1900
+ using CharType = int16_t;
+#else
+ using CharType = char16_t;
+#endif
+ using ConverterType = std::codecvt_utf8_utf16<CharType>;
+ std::wstring_convert<ConverterType, CharType> code_converter{};
+
+ std::basic_string<CharType> utf16_string;
+ try {
+ utf16_string = code_converter.from_bytes(string_start, string_end);
+ } catch (...) {
+ // In the case that any error occurs, just try returning a string in the
+ // user's current locale instead.
+ return mxCreateString(string_start);
+ }
+
+ // Store the converter UTF-16 string in a mxCharMatrix and return it.
+ const mwSize dimensions[2] = {1, utf16_string.size()};
+ mxArray* character_matrix = mxCreateCharArray(2, dimensions);
+ mxChar* character_matrix_pointer = mxGetChars(character_matrix);
+ std::copy(utf16_string.data(), utf16_string.data() + utf16_string.size(),
+ character_matrix_pointer);
+
+ return character_matrix;
+}
+
+} // namespace util
+} // namespace matlab
+} // namespace arrow
diff --git a/matlab/src/util/handle_status.h b/matlab/src/util/unicode_conversion.h
similarity index 66%
copy from matlab/src/util/handle_status.h
copy to matlab/src/util/unicode_conversion.h
index 18b0773..0a67b0e 100644
--- a/matlab/src/util/handle_status.h
+++ b/matlab/src/util/unicode_conversion.h
@@ -15,18 +15,20 @@
// specific language governing permissions and limitations
// under the License.
-#ifndef MLARROW_UTIL_HANDLE_STATUS_H
-#define MLARROW_UTIL_HANDLE_STATUS_H
+#ifndef ARROW_MATLAB_UTIL_UNICODE_CONVERSION_H
+#define ARROW_MATLAB_UTIL_UNICODE_CONVERSION_H
-#include <arrow/status.h>
+#include <string>
+#include <mex.h>
-namespace mlarrow {
+namespace arrow {
+namespace matlab {
namespace util {
-// Terminates execution and returns to the MATLAB prompt,
-// displaying an error message if the given status
-// indicates that an error has occurred.
-void HandleStatus(const arrow::Status& status);
+// Converts a UTF-8 encoded std::string to a heap-allocated UTF-16 encoded
+// mxCharArray.
+mxArray* ConvertUTF8StringToUTF16CharMatrix(const std::string& utf8_string);
} // namespace util
-} // namespace mlarrow
+} // namespace matlab
+} // namespace arrow
-#endif // MLARROW_UTIL_HANDLE_STATUS_H
+#endif /* ARROW_MATLAB_UTIL_UNICODE_CONVERSION_H */
diff --git a/matlab/test/corrupted_feather_file.feather b/matlab/test/corrupted_feather_file.feather
deleted file mode 100755
index 84c5a5e..0000000
--- a/matlab/test/corrupted_feather_file.feather
+++ /dev/null
@@ -1,5 +0,0 @@
-FEW1 ���> �@UUUUUU�? @
-
- � t � � 4 � � P Z��� uint64 ��� ` ���� uint32 P��� X ���� uint16 ���� P 2��� uint8 ���� H z��� int64 (��� 8 ���� int32 p��� 0
-��� int16 ���� ( R��� , int8 $ ���� , double "
- ( X FEW1
\ No newline at end of file
diff --git a/matlab/test/not_a_feather_file.feather b/matlab/test/not_a_feather_file.feather
deleted file mode 100755
index 59f6078..0000000
--- a/matlab/test/not_a_feather_file.feather
+++ /dev/null
@@ -1,18 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-This is not a Feather file.
diff --git a/matlab/test/numeric_datatypes_6th_variable_name_is_empty.feather b/matlab/test/numeric_datatypes_6th_variable_name_is_empty.feather
deleted file mode 100755
index 6d6dcd6..0000000
Binary files a/matlab/test/numeric_datatypes_6th_variable_name_is_empty.feather and /dev/null differ
diff --git a/matlab/test/numeric_datatypes_with_nan_column.feather b/matlab/test/numeric_datatypes_with_nan_column.feather
deleted file mode 100755
index 4da6184..0000000
Binary files a/matlab/test/numeric_datatypes_with_nan_column.feather and /dev/null differ
diff --git a/matlab/test/numeric_datatypes_with_nan_row.feather b/matlab/test/numeric_datatypes_with_nan_row.feather
deleted file mode 100755
index 65a0b50..0000000
Binary files a/matlab/test/numeric_datatypes_with_nan_row.feather and /dev/null differ
diff --git a/matlab/test/numeric_datatypes_with_no_nulls.feather b/matlab/test/numeric_datatypes_with_no_nulls.feather
deleted file mode 100755
index 37ea6bf..0000000
Binary files a/matlab/test/numeric_datatypes_with_no_nulls.feather and /dev/null differ
diff --git a/matlab/test/tfeather.m b/matlab/test/tfeather.m
new file mode 100755
index 0000000..625a3a5
--- /dev/null
+++ b/matlab/test/tfeather.m
@@ -0,0 +1,232 @@
+classdef tfeather < matlab.unittest.TestCase
+ % Tests for MATLAB featherread and featherwrite.
+
+ % Licensed to the Apache Software Foundation (ASF) under one or more
+ % contributor license agreements. See the NOTICE file distributed with
+ % this work for additional information regarding copyright ownership.
+ % The ASF licenses this file to you under the Apache License, Version
+ % 2.0 (the "License"); you may not use this file except in compliance
+ % with the License. You may obtain a copy of the License at
+ %
+ % http://www.apache.org/licenses/LICENSE-2.0
+ %
+ % Unless required by applicable law or agreed to in writing, software
+ % distributed under the License is distributed on an "AS IS" BASIS,
+ % WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ % implied. See the License for the specific language governing
+ % permissions and limitations under the License.
+
+ methods(TestClassSetup)
+
+ function addFeatherFunctionsToMATLABPath(testCase)
+ import matlab.unittest.fixtures.PathFixture
+ % Add Feather test utilities to the MATLAB path.
+ testCase.applyFixture(PathFixture('util'));
+ % Add featherread and featherwrite to the MATLAB path.
+ testCase.applyFixture(PathFixture(fullfile('..', 'src')));
+ % featherreadmex must be on the MATLAB path.
+ testCase.assertTrue(~isempty(which('featherreadmex')), ...
+ '''featherreadmex'' must be on the MATLAB path. Use ''addpath'' to add folders to the MATLAB path.');
+ % featherwritemex must be on the MATLAB path.
+ testCase.assertTrue(~isempty(which('featherwritemex')), ...
+ '''featherwritemex'' must be on to the MATLAB path. Use ''addpath'' to add folders to the MATLAB path.');
+ end
+
+ end
+
+ methods(TestMethodSetup)
+
+ function setupTempWorkingDirectory(testCase)
+ import matlab.unittest.fixtures.WorkingFolderFixture;
+ testCase.applyFixture(WorkingFolderFixture);
+ end
+
+ end
+
+ methods(Test)
+
+ function NumericDatatypesNoNulls(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = createTable;
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ function NumericDatatypesWithNaNRow(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ t = createTable;
+
+ variableNames = {'single', ...
+ 'double', ...
+ 'int8', ...
+ 'int16', ...
+ 'int32', ...
+ 'int64', ...
+ 'uint8', ...
+ 'uint16', ...
+ 'uint32', ...
+ 'uint64'};
+ variableTypes = repmat({'double'}, 10, 1)';
+ numRows = 1;
+ numVariables = 10;
+
+ addRow = table('Size', [numRows, numVariables], ...
+ 'VariableTypes', variableTypes, ...
+ 'VariableNames', variableNames);
+ addRow(1,:) = {NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN};
+ actualTable = [t; addRow];
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ function NumericDatatypesWithNaNColumns(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = createTable;
+ actualTable.double = [NaN; NaN; NaN];
+ actualTable.int64 = [NaN; NaN; NaN];
+
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ function NumericDatatypesWithExpInfSciNotation(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = createTable;
+ actualTable.single(2) = 1.0418e+06;
+
+ actualTable.double(1) = Inf;
+ actualTable.double(2) = exp(9);
+
+ actualTable.int64(2) = 1.0418e+03;
+
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ function IgnoreRowVarNames(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = createTable;
+ time = {'day1', 'day2', 'day3'};
+ actualTable.Properties.RowNames = time;
+ expectedTable = featherRoundTrip(filename, actualTable);
+ actualTable = createTable;
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ function NotFeatherExtension(testCase)
+ filename = fullfile(pwd, 'temp.txt');
+
+ actualTable = createTable;
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ function EmptyTable(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = table;
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ function zeroByNTable(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = createTable;
+ actualTable([1, 2], :) = [];
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyEqual(actualTable, expectedTable);
+ end
+
+ % %%%%%%%%%%%%%%%%%%%
+ % Negative test cases
+ % %%%%%%%%%%%%%%%%%%%
+
+ function ErrorIfUnableToOpenFile(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ testCase.verifyError(@() featherread(filename), 'MATLAB:arrow:UnableToOpenFile');
+ end
+
+ function ErrorIfCorruptedFeatherFile(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ t = createTable;
+ featherwrite(filename, t);
+
+ fileID = fopen(filename, 'w');
+ fwrite(fileID, [1; 5]);
+ fclose(fileID);
+
+ testCase.verifyError(@() featherread(filename), 'MATLAB:arrow:status:Invalid');
+ end
+
+ function ErrorIfInvalidFilenameDatatype(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ t = createTable;
+
+ testCase.verifyError(@() featherwrite({filename}, t), 'MATLAB:arrow:InvalidFilenameDatatype');
+ testCase.verifyError(@() featherread({filename}), 'MATLAB:arrow:InvalidFilenameDatatype');
+ end
+
+ function ErrorIfTooManyInputs(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ t = createTable;
+
+ testCase.verifyError(@() featherwrite(filename, t, 'SomeValue', 'SomeOtherValue'), 'MATLAB:TooManyInputs');
+ testCase.verifyError(@() featherread(filename, 'SomeValue', 'SomeOtherValue'), 'MATLAB:TooManyInputs');
+ end
+
+ function ErrorIfTooFewInputs(testCase)
+ testCase.verifyError(@() featherwrite(), 'MATLAB:narginchk:notEnoughInputs');
+ testCase.verifyError(@() featherread(), 'MATLAB:narginchk:notEnoughInputs');
+ end
+
+ function ErrorIfMultiColVarExist(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ age = [38; 43; 38; 40; 49];
+ smoker = logical([1; 0; 1; 0; 1]);
+ height = [71; 69; 64; 67; 64];
+ weight = [176; 163; 131; 133; 119];
+ bloodPressure = [124, 93; 109, 77; 125, 83; 117, 75; 122, 80];
+
+ t = table(age, smoker, height, weight, bloodPressure);
+
+ testCase.verifyError(@() featherwrite(filename, t), 'MATLAB:arrow:UnsupportedVariableType');
+ end
+
+ function UnsupportedMATLABDatatypes(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = createTable;
+ calendarDurationVariable = [calendarDuration(1, 7, 9); ...
+ calendarDuration(2, 1, 1); ...
+ calendarDuration(5, 3, 2)];
+ actualTable = addvars(actualTable, calendarDurationVariable);
+
+ testCase.verifyError(@() featherwrite(filename, actualTable) ,'MATLAB:arrow:UnsupportedVariableType');
+ end
+
+ function NumericComplexUnsupported(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ actualTable = createTable;
+ actualTable.single(1) = 1.0418 + 2i;
+ actualTable.double(2) = exp(9) + 5i;
+ actualTable.int64(2) = 1.0418e+03;
+
+ expectedTable = featherRoundTrip(filename, actualTable);
+ testCase.verifyNotEqual(actualTable, expectedTable);
+ end
+
+ end
+
+end
diff --git a/matlab/test/tfeathermex.m b/matlab/test/tfeathermex.m
new file mode 100644
index 0000000..fa79b4b
--- /dev/null
+++ b/matlab/test/tfeathermex.m
@@ -0,0 +1,76 @@
+classdef tfeathermex < matlab.unittest.TestCase
+ % Tests for MATLAB featherreadmex and featherwritemex.
+
+ % Licensed to the Apache Software Foundation (ASF) under one or more
+ % contributor license agreements. See the NOTICE file distributed with
+ % this work for additional information regarding copyright ownership.
+ % The ASF licenses this file to you under the Apache License, Version
+ % 2.0 (the "License"); you may not use this file except in compliance
+ % with the License. You may obtain a copy of the License at
+ %
+ % http://www.apache.org/licenses/LICENSE-2.0
+ %
+ % Unless required by applicable law or agreed to in writing, software
+ % distributed under the License is distributed on an "AS IS" BASIS,
+ % WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ % implied. See the License for the specific language governing
+ % permissions and limitations under the License.
+
+ methods(TestClassSetup)
+
+ function addFeatherFunctionsToMATLABPath(testCase)
+ import matlab.unittest.fixtures.PathFixture
+ % Add Feather test utilities to the MATLAB path.
+ testCase.applyFixture(PathFixture('util'));
+ % Add featherread and featherwrite to the MATLAB path.
+ testCase.applyFixture(PathFixture(fullfile('..', 'src')));
+ % featherreadmex must be on the MATLAB path.
+ testCase.assertTrue(~isempty(which('featherreadmex')), ...
+ '''featherreadmex'' must be on the MATLAB path. Use ''addpath'' to add folders to the MATLAB path.');
+ % featherwritemex must be on the MATLAB path.
+ testCase.assertTrue(~isempty(which('featherwritemex')), ...
+ '''featherwritemex'' must be on to the MATLAB path. Use ''addpath'' to add folders to the MATLAB path.');
+ end
+
+ end
+
+ methods(TestMethodSetup)
+
+ function setupTempWorkingDirectory(testCase)
+ import matlab.unittest.fixtures.WorkingFolderFixture;
+ testCase.applyFixture(WorkingFolderFixture);
+ end
+
+ end
+
+ methods(Test)
+
+ function NumericDatatypesNulls(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ [expectedVariables, expectedMetadata] = createVariablesAndMetadataStructs();
+ [actualVariables, ~] = featherMEXRoundTrip(filename, expectedVariables, expectedMetadata);
+ testCase.verifyEqual([actualVariables.Valid], [expectedVariables.Valid]);
+ end
+
+ function InvalidMATLABTableVariableNames(testCase)
+ filename = fullfile(pwd, 'temp.feather');
+
+ % Create a table with an invalid MATLAB table variable name.
+ invalidVariable = mlarrow.util.createVariableStruct('double', 1, true, '@');
+ validVariable = mlarrow.util.createVariableStruct('double', 1, true, 'Valid');
+ variables = [invalidVariable, validVariable];
+ metadata = mlarrow.util.createMetadataStruct('', 1, 2);
+ featherwritemex(filename, variables, metadata);
+ t = featherread(filename);
+
+ testCase.verifyEqual(t.Properties.VariableNames{1}, 'x_');
+ testCase.verifyEqual(t.Properties.VariableNames{2}, 'Valid');
+
+ testCase.verifyEqual(t.Properties.VariableDescriptions{1}, 'Original variable name: ''@''');
+ testCase.verifyEqual(t.Properties.VariableDescriptions{2}, '');
+ end
+
+ end
+
+end
diff --git a/matlab/test/tfeatherread.m b/matlab/test/tfeatherread.m
deleted file mode 100755
index 3e633a6..0000000
--- a/matlab/test/tfeatherread.m
+++ /dev/null
@@ -1,141 +0,0 @@
-classdef tfeatherread < matlab.unittest.TestCase
- % Tests for MATLAB featherread.
-
- % Licensed to the Apache Software Foundation (ASF) under one or more
- % contributor license agreements. See the NOTICE file distributed with
- % this work for additional information regarding copyright ownership.
- % The ASF licenses this file to you under the Apache License, Version
- % 2.0 (the "License"); you may not use this file except in compliance
- % with the License. You may obtain a copy of the License at
- %
- % http://www.apache.org/licenses/LICENSE-2.0
- %
- % Unless required by applicable law or agreed to in writing, software
- % distributed under the License is distributed on an "AS IS" BASIS,
- % WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
- % implied. See the License for the specific language governing
- % permissions and limitations under the License.
-
- methods(TestClassSetup)
-
- function addFeatherreadToMATLABPath(testcase)
- import matlab.unittest.fixtures.PathFixture
- % Add featherread.m to the MATLAB path.
- testcase.applyFixture(PathFixture('../'));
- end
-
- end
-
- methods(Test)
-
- function NumericDatatypesNoNulls(testCase)
- filename = 'numeric_datatypes_with_no_nulls.feather';
- actualTable = featherread(filename);
-
- variableNames = {'single', ...
- 'double', ...
- 'int8', ...
- 'int16', ...
- 'int32', ...
- 'int64', ...
- 'uint8', ...
- 'uint16', ...
- 'uint32', ...
- 'uint64'};
- variableTypes = {'single', ...
- 'double', ...
- 'int8', ...
- 'int16', ...
- 'int32', ...
- 'int64', ...
- 'uint8', ...
- 'uint16', ...
- 'uint32', ...
- 'uint64'};
- numRows = 2;
- numVariables = 10;
-
- expectedTable = table('Size', [numRows, numVariables], 'VariableTypes', variableTypes, 'VariableNames', variableNames);
- expectedTable(1, :) = {1/3, 2/3, 1, 2, 3, 4, 9, 10, 11, 12};
- expectedTable(2, :) = {4, 5, 5, 6, 7, 8, 13, 14, 15, 16};
-
- testCase.verifyEqual(actualTable, expectedTable);
- end
-
- function NumericDatatypesWithEmptyVariableName(testCase)
- filename = 'numeric_datatypes_6th_variable_name_is_empty.feather';
- t = featherread(filename);
-
- actualVariableName = t.Properties.VariableNames(6);
- expectedVariableName = {'x'};
- testCase.verifyEqual(actualVariableName, expectedVariableName);
- end
-
- function NumericDatatypesWithNaNRow(testCase)
- filename = 'numeric_datatypes_with_nan_row.feather';
- t = featherread(filename);
-
- actualVariableData = t{3, {'single'}};
- expectedVariableData = single(NaN);
- testCase.verifyEqual(actualVariableData, expectedVariableData);
-
- actualRemainingVariablesData = t{3, {'double','int8','int16','int32','int64',...
- 'uint8','uint16','uint32','uint64'}};
- expectedRemainingVariablesData = double([NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]);
- testCase.verifyEqual(actualRemainingVariablesData, expectedRemainingVariablesData);
- end
-
- function NumericDatatypesWithNaNColumn(testCase)
- filename = 'numeric_datatypes_with_nan_column.feather';
- t = featherread(filename);
-
- actualVariable6 = t.int64;
- expectedVariable6 = double([NaN; NaN]);
- testCase.verifyEqual(actualVariable6, expectedVariable6);
-
- actualVariable9 = t.uint32;
- expectedVariable9 = double([NaN;NaN]);
- testCase.verifyEqual(actualVariable9, expectedVariable9);
- end
-
- % %%%%%%%%%%%%%%%%%%%
- % Negative test cases
- % %%%%%%%%%%%%%%%%%%%
- function ErrorIfNotAFeatherFile(testCase)
- filename = 'not_a_feather_file.feather';
-
- testCase.verifyError(@() featherread(filename), 'MATLAB:arrow:status:Invalid');
- end
-
- function ErrorIfUnableToOpenFile(testCase)
- filename = 'nonexistent.feather';
-
- testCase.verifyError(@() featherread(filename), 'MATLAB:arrow:UnableToOpenFile');
- end
-
- function ErrorIfCorruptedFeatherFile(testCase)
- filename = 'corrupted_feather_file.feather';
-
- testCase.verifyError(@() featherread(filename), 'MATLAB:arrow:status:Invalid');
- end
-
- function ErrorIfInvalidFilenameDatatype(testCase)
- filename = {'numeric_datatypes_with_no_nulls.feather'};
-
- testCase.verifyError(@() featherread(filename), 'MATLAB:arrow:InvalidFilenameDatatype');
- end
-
- function ErroriIfTooManyInputs(testCase)
- filename = 'numeric_datatypes_with_nan_column.feather';
-
- testCase.verifyError(@() featherread(filename, 'SomeValue'), 'MATLAB:TooManyInputs');
- end
-
- function ErrorIfTooFewInputs(testCase)
- testCase.verifyError(@() featherread(), 'MATLAB:narginchk:notEnoughInputs');
- end
-
- end
-
-end
-
diff --git a/matlab/test/util/createTable.m b/matlab/test/util/createTable.m
new file mode 100644
index 0000000..2bf67c6
--- /dev/null
+++ b/matlab/test/util/createTable.m
@@ -0,0 +1,68 @@
+function t = createTable()
+% CREATETABLE Helper function for creating test table.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+variableNames = {'uint8', ...
+ 'uint16', ...
+ 'uint32', ...
+ 'uint64', ...
+ 'int8', ...
+ 'int16', ...
+ 'int32', ...
+ 'int64', ...
+ 'single', ...
+ 'double'};
+
+variableTypes = {'uint8', ...
+ 'uint16', ...
+ 'uint32', ...
+ 'uint64', ...
+ 'int8', ...
+ 'int16', ...
+ 'int32', ...
+ 'int64', ...
+ 'single', ...
+ 'double'};
+
+uint8Data = uint8([1; 2; 3]);
+uint16Data = uint16([1; 2; 3]);
+uint32Data = uint32([1; 2; 3]);
+uint64Data = uint64([1; 2; 3]);
+int8Data = int8([1; 2; 3]);
+int16Data = int16([1; 2; 3]);
+int32Data = int32([1; 2; 3]);
+int64Data = int64([1; 2; 3]);
+singleData = single([1/2; 1/4; 1/8]);
+doubleData = double([1/10; 1/100; 1/1000]);
+
+numRows = 3;
+numVariables = 10;
+
+t = table('Size', [numRows, numVariables], 'VariableTypes', variableTypes, 'VariableNames', variableNames);
+
+t.uint8 = uint8Data;
+t.uint16 = uint16Data;
+t.uint32 = uint32Data;
+t.uint64 = uint64Data;
+t.int8 = int8Data;
+t.int16 = int16Data;
+t.int32 = int32Data;
+t.int64 = int64Data;
+t.single = singleData;
+t.double = doubleData;
+
+end
\ No newline at end of file
diff --git a/matlab/test/util/createVariablesAndMetadataStructs.m b/matlab/test/util/createVariablesAndMetadataStructs.m
new file mode 100644
index 0000000..01a8f58
--- /dev/null
+++ b/matlab/test/util/createVariablesAndMetadataStructs.m
@@ -0,0 +1,98 @@
+function [variables, metadata] = createVariablesAndMetadataStructs()
+% CREATEVARIABLESANDMETADATASTRUCTS Helper function for creating
+% Feather MEX variables and metadata structs.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+import mlarrow.util.*;
+
+type = 'uint8';
+data = uint8([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'uint8';
+uint8Variable = createVariableStruct(type, data, valid, name);
+
+type = 'uint16';
+data = uint16([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'uint16';
+uint16Variable = createVariableStruct(type, data, valid, name);
+
+type = 'uint32';
+data = uint32([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'uint32';
+uint32Variable = createVariableStruct(type, data, valid, name);
+
+type = 'uint64';
+data = uint64([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'uint64';
+uint64Variable = createVariableStruct(type, data, valid, name);
+
+type = 'int8';
+data = int8([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'int8';
+int8Variable = createVariableStruct(type, data, valid, name);
+
+type = 'int16';
+data = int16([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'int16';
+int16Variable = createVariableStruct(type, data, valid, name);
+
+type = 'int32';
+data = int32([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'int32';
+int32Variable = createVariableStruct(type, data, valid, name);
+
+type = 'int64';
+data = int64([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'int64';
+int64Variable = createVariableStruct(type, data, valid, name);
+
+type = 'single';
+data = single([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'single';
+singleVariable = createVariableStruct(type, data, valid, name);
+
+type = 'double';
+data = double([1; 2; 3]);
+valid = logical([0; 1; 0]);
+name = 'double';
+doubleVariable = createVariableStruct(type, data, valid, name);
+
+variables = [uint8Variable, ...
+ uint16Variable, ...
+ uint32Variable, ...
+ uint64Variable, ...
+ int8Variable, ...
+ int16Variable, ...
+ int32Variable, ...
+ int64Variable, ...
+ singleVariable, ...
+ doubleVariable];
+
+description = 'test';
+numRows = 3;
+numVariables = length(variables);
+
+metadata = createMetadataStruct(description, numRows, numVariables);
+end
diff --git a/matlab/test/util/featherMEXRoundTrip.m b/matlab/test/util/featherMEXRoundTrip.m
new file mode 100644
index 0000000..49ab183
--- /dev/null
+++ b/matlab/test/util/featherMEXRoundTrip.m
@@ -0,0 +1,22 @@
+function [variablesOut, metadataOut] = featherMEXRoundTrip(filename, variablesIn, metadataIn)
+% FEATHERMEXROUNDTRIP Helper function for round tripping variables
+% and metadata structs to a Feather file.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+featherwritemex(filename, variablesIn, metadataIn);
+[variablesOut, metadataOut] = featherreadmex(filename);
+end
\ No newline at end of file
diff --git a/matlab/test/util/featherRoundTrip.m b/matlab/test/util/featherRoundTrip.m
new file mode 100644
index 0000000..18f8056
--- /dev/null
+++ b/matlab/test/util/featherRoundTrip.m
@@ -0,0 +1,22 @@
+function tableOut = featherRoundTrip(filename, tableIn)
+% FEATHERROUNDTRIP Helper function for round tripping a table
+% to a Feather file.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+
+featherwrite(filename, tableIn);
+tableOut = featherread(filename);
+end
\ No newline at end of file