You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/10/07 05:11:57 UTC
[GitHub] [arrow] emkornfield opened a new pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
emkornfield opened a new pull request #8374:
URL: https://github.com/apache/arrow/pull/8374
@kiszk @jacques-n @wesm @pitrou @BryanCutler @nealrichardson this capture my understanding of the mailing list conversation on endianness. Please let me know if I've mischaracterized anything (I'll do a proof reading/compiling round once as long as the general points are agreed upon).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r506060515
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
Review comment:
rephrased to account for popularity of endiannes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#issuecomment-709814489
I'm having trouble getting the source build environment setup on my machine to verify correct layout, I'll try again tomorrow. Let me know if there are other comments otherwise I think we can merge this and go ahead and start incorporating the other PRs for endianness.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#issuecomment-704700018
https://issues.apache.org/jira/browse/ARROW-10203
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501356855
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
Review comment:
Robu`s`t
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501448247
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
+
+1. C++
+
+Implementations that do not intend to implement cross
+
+1. Java
Review comment:
I see. Before the asking review for the PR, I will gather the consensus on the ML.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r506060391
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500762410
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
Review comment:
On some platforms (e.g. C++), there are additional spaces to support features (e.g. Parquet support in C++)
Can we create the third level (e.g. misc. or library dependant)? Or do we put these works into the second level?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501447270
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
+
+1. C++
+
+Implementations that do not intend to implement cross
+
+1. Java
Review comment:
I think @jacques-n was against cross-platform support in Java?
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
Review comment:
I would actually place that in level #1. I meant level #2 applies only to Arrow specific transport technologies (IPC format and flight)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r507708717
##########
File path: docs/source/python/ipc.rst
##########
@@ -330,21 +330,3 @@ An object can be reconstructed from its component-based representation using
``deserialize_components`` is also available as a method on
``SerializationContext`` objects.
-
-Serializing pandas Objects
Review comment:
Had to remove this to build the docs. This is a deprecated functionality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501448247
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
+
+1. C++
+
+Implementations that do not intend to implement cross
+
+1. Java
Review comment:
I see. Before the asking review for the PR, I will gather the consensus on the ML.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#issuecomment-712125394
+1, will merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500762410
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
Review comment:
On some platforms (e.g. C++), there are additional works (e.g. Parquet support in C++)
Can we create the third level? Or do we put these works into the second level?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500766332
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
+
+1. C++
+
+Implementations that do not intend to implement cross
+
+1. Java
Review comment:
Just FYI: While there is no PR to implement cross platform support for Java, I will submit the PR later after supporting the level 1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500762410
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
Review comment:
On some platforms (e.g. C++), there are additional works (e.g. Parquet support in C++)
Can we create the third level (e.g. misc.)? Or do we put these works into the second level?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501874856
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
Review comment:
this may need an empy line to be formatted as items.
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
Review comment:
this may need an empy line to be formatted as items.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500762410
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
Review comment:
On some platforms (e.g. C++), there are additional works (e.g. Parquet support in C++)
Can we create the third level (e.g. misc. or library dependant)? Or do we put these works into the second level?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] BryanCutler commented on a change in pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r506618937
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,45 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+The Arrow format allows setting endianness. Due to the popularity of little endian
+architectures most of implementation assume little endian by default. There has been some
+effort to support big endian platforms as well. Based on a mailing list discussion,
Review comment:
Would it be good to link some JIRAs either here or below?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501874856
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
Review comment:
this may need an empy line to be formatted as items.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500877701
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
Review comment:
I wouldn't say that. Arrow lets you specify the endianness. But many computer systems these days (most?) are little-endian and most Arrow-based applications probably run on little-endian systems.
(note "systems": e.g. PPC CPUs can be configured in both LE and BE modes, but under Linux they use LE)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r506060257
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
Review comment:
Clarified in doc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #8374: ARROW-10203: [Doc] Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #8374:
URL: https://github.com/apache/arrow/pull/8374
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501874856
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
Review comment:
this may need an empy line to be formatted as items.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500766510
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
+
+1. C++
+
+Implementations that do not intend to implement cross
Review comment:
`cross` -> `cross platform support`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors …
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r500765879
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
Review comment:
I think that it is easy to read if we start a new paragraph with `The current implementations ...`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501447504
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
Review comment:
I would actually place that in level #1. I meant level #2 applies only to Arrow specific transport technologies (IPC format and flight)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #8374: ARROW-10203: Give guidance on big-endian support in the contributors docs
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #8374:
URL: https://github.com/apache/arrow/pull/8374#discussion_r501447270
##########
File path: docs/source/developers/contributing.rst
##########
@@ -304,3 +304,40 @@ to your branch, which they sometimes do to help move a pull request along.
In addition, the GitHub PR "suggestion" feature can also add commits to
your branch, so it is possible that your local copy of your branch is missing
some additions.
+
+Guidance for specific features
+==============================
+
+From time to time the community has discussions on specific types of features
+and improvements that they expect to support. This section outlines decisions
+that have been made in this regard.
+
+Endianess
++++++++++
+Arrow is primarily a little endian format there has been some effort to
+support big endian platforms as well. Based on a mailing list discussion,
+The requirements for a new platform are:
+
+1. A robut (non-flaky, returns results in a reasonable time) Continuous integration setup.
+2. Performance benchmarks in performance critical parts of the code to demonstrate no
+ regression.
+
+Furthermore for big-endianess support there are two levels that an implementation can support
+1. Native endianness (all arrow communication happens with processes of the same endianness.
+2. Cross platform support (implementations will do byte reordering when appropriate for IPC
+ and flight messages).
+
+The decision on what level to support is based on maintainers preferences for complexity and
+technical risk. In general all implementations should be open to native endianness support
+(provided the CI and performance requirements are met). Cross endianness support is a question
+for individual maintainers. The current implementations aiming for cross platform support are:
+
+1. C++
+
+Implementations that do not intend to implement cross
+
+1. Java
Review comment:
I think @jacques-n was against cross-platform support in Java?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org