You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/22 16:39:55 UTC
[GitHub] [arrow] eerhardt opened a new pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
eerhardt opened a new pull request #10973:
URL: https://github.com/apache/arrow/pull/10973
cc @pitrou - if you want to take a first look. I'm not sure how to get the tests running locally yet, but I've run the integration app a few times against the 'primitive.json' file to test it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-927373012
I finally completed the initial work here to test the C# library in the integration tests. I'm not certain whom I should tag for a review of the infra changes I made here. @pitrou @kszucs - can you suggest the appropriate people?
Also tagging recent contributors to the C# library, in case they have any feedback:
@pgovind @suhsteve @chutchinson @Ulimo @mr-smidge @zgramana @HashidaTKS @nhustler @0x0L @llerrowa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-908308624
> cc @pitrou - if you want to take a first look. I'm not sure how to get the tests running locally yet, but I've run the integration app a few times against the 'primitive.json' file to test it out.
Sorry for the delay. If you're talking about the integration tests as they are run in CI, you can run them using `archery docker run conda-integration`. See https://arrow.apache.org/docs/developers/archery.html for installing Archery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-908771615
Thanks for the help @pitrou! I'll keep working on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-908311354
You may also temporarily tweak `ci/scripts/integration_arrow.sh` to disable some implementations, if you want to iterate faster.
(since C++ is the most featureful implementation, you could for example disable Java, Javascript, Go and Rust)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a change in pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on a change in pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#discussion_r717072995
##########
File path: dev/archery/archery/integration/datagen.py
##########
@@ -1557,82 +1557,102 @@ def _temp_path():
generate_primitive_case([0, 0, 0], name='primitive_zerolength'),
generate_primitive_large_offsets_case([17, 20])
+ .skip_category('csharp')
.skip_category('Go')
.skip_category('JS'),
generate_null_case([10, 0])
+ .skip_category('csharp')
+ .skip_category('Go') # TODO(ARROW-7901)
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_null_trivial_case([0, 0])
+ .skip_category('csharp')
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_decimal128_case()
.skip_category('Rust'),
generate_decimal256_case()
+ .skip_category('csharp')
Review comment:
It was pretty simple to implement this now, so I did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a change in pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on a change in pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#discussion_r717004347
##########
File path: dev/archery/archery/cli.py
##########
@@ -692,6 +692,8 @@ def _set_default(opt, default):
help="Seed for PRNG when generating test data")
@click.option('--with-cpp', type=bool, default=False,
help='Include C++ in integration tests')
+@click.option('--with-csharp', type=bool, default=False,
+ help='Include CSharp in integration tests')
Review comment:
Yes, good call.
##########
File path: dev/archery/archery/integration/datagen.py
##########
@@ -1557,82 +1557,102 @@ def _temp_path():
generate_primitive_case([0, 0, 0], name='primitive_zerolength'),
generate_primitive_large_offsets_case([17, 20])
+ .skip_category('csharp')
.skip_category('Go')
.skip_category('JS'),
generate_null_case([10, 0])
+ .skip_category('csharp')
+ .skip_category('Go') # TODO(ARROW-7901)
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_null_trivial_case([0, 0])
+ .skip_category('csharp')
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_decimal128_case()
.skip_category('Rust'),
generate_decimal256_case()
+ .skip_category('csharp')
Review comment:
It is supported. However, I needed to cut things off somewhere in order to get the initial implementation out. If it is easy to implement now that I have everything working, I'll implement it.
##########
File path: dev/archery/archery/integration/tester_csharp.py
##########
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+from .tester import Tester
+from .util import run_cmd, ARROW_ROOT_DEFAULT, log
+
+
+class CSharpTester(Tester):
+ PRODUCER = True
+ CONSUMER = True
+
+ EXE_PATH = os.path.join(
+ ARROW_ROOT_DEFAULT, 'csharp/artifacts/Apache.Arrow.IntegrationTest/Debug/netcoreapp3.1/Apache.Arrow.IntegrationTest')
+
+ name = 'csharp'
Review comment:
Yes, good call.
##########
File path: dev/archery/archery/integration/datagen.py
##########
@@ -1557,82 +1557,102 @@ def _temp_path():
generate_primitive_case([0, 0, 0], name='primitive_zerolength'),
generate_primitive_large_offsets_case([17, 20])
+ .skip_category('csharp')
.skip_category('Go')
.skip_category('JS'),
generate_null_case([10, 0])
+ .skip_category('csharp')
+ .skip_category('Go') # TODO(ARROW-7901)
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_null_trivial_case([0, 0])
+ .skip_category('csharp')
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_decimal128_case()
.skip_category('Rust'),
generate_decimal256_case()
+ .skip_category('csharp')
Review comment:
It was pretty simple to implement this now, so I did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#discussion_r716714655
##########
File path: dev/archery/archery/cli.py
##########
@@ -692,6 +692,8 @@ def _set_default(opt, default):
help="Seed for PRNG when generating test data")
@click.option('--with-cpp', type=bool, default=False,
help='Include C++ in integration tests')
+@click.option('--with-csharp', type=bool, default=False,
+ help='Include CSharp in integration tests')
Review comment:
Should it be spelled "C#" in this message?
##########
File path: dev/archery/archery/integration/datagen.py
##########
@@ -1557,82 +1557,102 @@ def _temp_path():
generate_primitive_case([0, 0, 0], name='primitive_zerolength'),
generate_primitive_large_offsets_case([17, 20])
+ .skip_category('csharp')
.skip_category('Go')
.skip_category('JS'),
generate_null_case([10, 0])
+ .skip_category('csharp')
+ .skip_category('Go') # TODO(ARROW-7901)
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_null_trivial_case([0, 0])
+ .skip_category('csharp')
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_decimal128_case()
.skip_category('Rust'),
generate_decimal256_case()
+ .skip_category('csharp')
Review comment:
According to https://github.com/apache/arrow/blob/master/docs/source/status.rst#data-types, Decimal256 should be supported?
##########
File path: dev/archery/archery/integration/tester_csharp.py
##########
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+from .tester import Tester
+from .util import run_cmd, ARROW_ROOT_DEFAULT, log
+
+
+class CSharpTester(Tester):
+ PRODUCER = True
+ CONSUMER = True
+
+ EXE_PATH = os.path.join(
+ ARROW_ROOT_DEFAULT, 'csharp/artifacts/Apache.Arrow.IntegrationTest/Debug/netcoreapp3.1/Apache.Arrow.IntegrationTest')
+
+ name = 'csharp'
Review comment:
Wouldn't "C#" be a better name for display?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a change in pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on a change in pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#discussion_r717004347
##########
File path: dev/archery/archery/cli.py
##########
@@ -692,6 +692,8 @@ def _set_default(opt, default):
help="Seed for PRNG when generating test data")
@click.option('--with-cpp', type=bool, default=False,
help='Include C++ in integration tests')
+@click.option('--with-csharp', type=bool, default=False,
+ help='Include CSharp in integration tests')
Review comment:
Yes, good call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a change in pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on a change in pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#discussion_r717007372
##########
File path: dev/archery/archery/integration/tester_csharp.py
##########
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+from .tester import Tester
+from .util import run_cmd, ARROW_ROOT_DEFAULT, log
+
+
+class CSharpTester(Tester):
+ PRODUCER = True
+ CONSUMER = True
+
+ EXE_PATH = os.path.join(
+ ARROW_ROOT_DEFAULT, 'csharp/artifacts/Apache.Arrow.IntegrationTest/Debug/netcoreapp3.1/Apache.Arrow.IntegrationTest')
+
+ name = 'csharp'
Review comment:
Yes, good call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-930264675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-908308624
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a change in pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on a change in pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#discussion_r717005085
##########
File path: dev/archery/archery/integration/datagen.py
##########
@@ -1557,82 +1557,102 @@ def _temp_path():
generate_primitive_case([0, 0, 0], name='primitive_zerolength'),
generate_primitive_large_offsets_case([17, 20])
+ .skip_category('csharp')
.skip_category('Go')
.skip_category('JS'),
generate_null_case([10, 0])
+ .skip_category('csharp')
+ .skip_category('Go') # TODO(ARROW-7901)
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_null_trivial_case([0, 0])
+ .skip_category('csharp')
.skip_category('Go') # TODO(ARROW-7901)
.skip_category('JS'), # TODO(ARROW-7900)
generate_decimal128_case()
.skip_category('Rust'),
generate_decimal256_case()
+ .skip_category('csharp')
Review comment:
It is supported. However, I needed to cut things off somewhere in order to get the initial implementation out. If it is easy to implement now that I have everything working, I'll implement it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-930264675
I'll try to take a second look quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-930240565
To move forward, I'll merge this either late today or tomorrow. Let me know if you wanted to review it and need more time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt closed pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt closed pull request #10973:
URL: https://github.com/apache/arrow/pull/10973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-908771615
Thanks for the help @pitrou! I'll keep working on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-908310089
Ah, you'll need to apply the following patch:
```diff
diff --git a/ci/scripts/integration_arrow.sh b/ci/scripts/integration_arrow.sh
index 453561d31..30cbb2d63 100755
--- a/ci/scripts/integration_arrow.sh
+++ b/ci/scripts/integration_arrow.sh
@@ -28,6 +28,7 @@ pip install -e $arrow_dir/dev/archery
archery integration \
--run-flight \
--with-cpp=1 \
+ --with-csharp=1 \
--with-java=1 \
--with-js=1 \
--with-go=1 \
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-903296102
https://issues.apache.org/jira/browse/ARROW-13689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-930321443
No objection from me. Thanks _a lot_ for working on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt commented on pull request #10973:
URL: https://github.com/apache/arrow/pull/10973#issuecomment-930240565
To move forward, I'll merge this either late today or tomorrow. Let me know if you wanted to review it and need more time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt closed pull request #10973: ARROW-13689: [C#][Integration] Initial commit of C# Integration tests
Posted by GitBox <gi...@apache.org>.
eerhardt closed pull request #10973:
URL: https://github.com/apache/arrow/pull/10973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org