You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/04 18:07:01 UTC
[GitHub] [arrow-rs] alamb opened a new pull request, #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
alamb opened a new pull request, #2324:
URL: https://github.com/apache/arrow-rs/pull/2324
# Which issue does this PR close?
N/A
# Rationale for this change
The parquet crate has a non trivial number of feature flags. Let's consistently document the flags in crates.io and link to them in docs.rs
https://crates.io/crates/arrow
https://crates.io/crates/parquet
Also while I was messing around I found some other places where the docs could be improved so I did.
# What changes are included in this PR?
1. Document parquet feature flags
2. Misc wording improvements
# Are there any user-facing changes?
readme is different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] ursabot commented on pull request #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #2324:
URL: https://github.com/apache/arrow-rs/pull/2324#issuecomment-1206199663
Benchmark runs are scheduled for baseline = 297a8faf535fd1189a92051d043bdd3990b046ba and contender = 8e30d06ea939dc8a3bdac8dfe366348e864b8dba. 8e30d06ea939dc8a3bdac8dfe366348e864b8dba is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/2133518e531c4bd5a23abb626948aee6...69b3339660ef426289ff724a8bda0eb7/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/8f5f42fd5f234d509d874719b67e241b...d7ce4db8a4034a16b05e527dcd05457f/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7257051d71f54fbc892ac22990ace648...8dee64b972394c30ac0f59a6ca5034b1/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/0a88c8d1117e406cb78c5faf78c42712...3b427f888c8b4611ac9154db5eff589a/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter commented on pull request #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #2324:
URL: https://github.com/apache/arrow-rs/pull/2324#issuecomment-1205625376
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/2324?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#2324](https://codecov.io/gh/apache/arrow-rs/pull/2324?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (175e80e) into [master](https://codecov.io/gh/apache/arrow-rs/commit/d87f6a456330f0eff00541b38017e8a982bf8033?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d87f6a4) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #2324 +/- ##
==========================================
- Coverage 80.53% 80.53% -0.01%
==========================================
Files 248 248
Lines 59971 59971
==========================================
- Hits 48296 48295 -1
- Misses 11675 11676 +1
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/2324?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [parquet\_derive/src/parquet\_field.rs](https://codecov.io/gh/apache/arrow-rs/pull/2324/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldF9kZXJpdmUvc3JjL3BhcnF1ZXRfZmllbGQucnM=) | `65.75% <0.00%> (-0.23%)` | :arrow_down: |
:mega: Codecov can now indicate which changes are the most critical in Pull Requests. [Learn more](https://about.codecov.io/product/feature/runtime-insights/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2324:
URL: https://github.com/apache/arrow-rs/pull/2324#discussion_r938681874
##########
arrow/README.md:
##########
@@ -55,25 +64,25 @@ Arrow seeks to uphold the Rust Soundness Pledge as articulated eloquently [here]
Where soundness in turn is defined as:
-> Code is unable to trigger undefined behaviour using safe APIs
+> Code is unable to trigger undefined behavior using safe APIs
Review Comment:
lol I am "correcting" the spelling (or maybe I can claim I am "standardi**z**ing" the spelling 😆
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2324:
URL: https://github.com/apache/arrow-rs/pull/2324#discussion_r938584363
##########
arrow/README.md:
##########
@@ -101,16 +110,38 @@ cargo run --example read_csv
[arrow]: https://arrow.apache.org/
+## Performance Tips
-## Performance
+Arrow aims to be as fast as possible out of the box, whilst not compromising on safety. However,
+it relies heavily on LLVM auto-vectorisation to achieve this. Unfortunately the LLVM defaults,
+particularly for x86_64, favour portability over performance, and LLVM will consequently avoid
+using more recent instructions that would result in errors on older CPUs.
-Most of the compute kernels benefit a lot from being optimized for a specific CPU target.
-This is especially so on x86-64 since without specifying a target the compiler can only assume support for SSE2 vector instructions.
-One of the following values as `-Ctarget-cpu=value` in `RUSTFLAGS` can therefore improve performance significantly:
+To address this it is recommended that you specify the override the LLVM defaults either
Review Comment:
Oops this keeps a typo from my PR, "specify the override the". Will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold merged pull request #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
Posted by GitBox <gi...@apache.org>.
tustvold merged PR #2324:
URL: https://github.com/apache/arrow-rs/pull/2324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2324:
URL: https://github.com/apache/arrow-rs/pull/2324#discussion_r938581821
##########
arrow/README.md:
##########
@@ -55,25 +64,25 @@ Arrow seeks to uphold the Rust Soundness Pledge as articulated eloquently [here]
Where soundness in turn is defined as:
-> Code is unable to trigger undefined behaviour using safe APIs
+> Code is unable to trigger undefined behavior using safe APIs
Review Comment:
... I shall British-ify the spelling one comment at a time 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2324: [Minor] Improve arrow and parquet READMEs, document parquet feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2324:
URL: https://github.com/apache/arrow-rs/pull/2324#discussion_r938112490
##########
arrow/src/lib.rs:
##########
@@ -18,41 +18,8 @@
//! A complete, safe, native Rust implementation of [Apache Arrow](https://arrow.apache.org), a cross-language
//! development platform for in-memory data.
//!
-//! # Performance Tips
Review Comment:
I found this section was just introduced by @tustvold in https://github.com/apache/arrow-rs/pull/2305 but it turns out there was already a section at the end of the README.md document that had the same information. I consolidated them in README.md and added a link "please see crates.io page" here
##########
arrow/README.md:
##########
@@ -101,16 +110,38 @@ cargo run --example read_csv
[arrow]: https://arrow.apache.org/
+## Performance Tips
-## Performance
+Arrow aims to be as fast as possible out of the box, whilst not compromising on safety. However,
Review Comment:
This content was moved from what was added in https://github.com/apache/arrow-rs/pull/2305
##########
arrow/README.md:
##########
@@ -34,18 +37,24 @@ The arrow crate follows the [SemVer standard](https://doc.rust-lang.org/cargo/re
However, for historical reasons, this crate uses versions with major numbers greater than `0.x` (e.g. `19.0.0`), unlike many other crates in the Rust ecosystem which spend extended time releasing versions `0.x` to signal planned ongoing API changes. Minor arrow releases contain only compatible changes, while major releases may contain breaking API changes.
-## Features
+## Feature Flags
-The arrow crate provides the following features which may be enabled:
+The `arrow` crate provides the following features which may be enabled in your `Cargo.toml`:
- `csv` (default) - support for reading and writing Arrow arrays to/from csv files
- `ipc` (default) - support for the [arrow-flight](https://crates.io/crates/arrow-flight) IPC and wire format
- `prettyprint` - support for formatting record batches as textual columns
- `js` - support for building arrow for WebAssembly / JavaScript
-- `simd` - (_Requires Nightly Rust_) alternate optimized
+- `simd` - (_Requires Nightly Rust_) Use alternate hand optimized
implementations of some [compute](https://github.com/apache/arrow-rs/tree/master/arrow/src/compute/kernels)
- kernels using explicit SIMD instructions available through [packed_simd_2](https://docs.rs/packed_simd_2/latest/packed_simd_2/).
+ kernels using explicit SIMD instructions via [packed_simd_2](https://docs.rs/packed_simd_2/latest/packed_simd_2/).
- `chrono-tz` - support of parsing timezone using [chrono-tz](https://docs.rs/chrono-tz/0.6.0/chrono_tz/)
+- `ffi` - bindings for the Arrow C [C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html)
Review Comment:
newly added feature flags
##########
arrow/src/compute/README.md:
##########
@@ -33,16 +33,16 @@ We use the term "kernel" to refer to particular general operation that contains
Types of functions
-* Scalar functions: elementwise functions that perform scalar operations in a
+- Scalar functions: elementwise functions that perform scalar operations in a
Review Comment:
the `prettier` tool did this
##########
parquet/README.md:
##########
@@ -19,17 +19,38 @@
# Apache Parquet Official Native Rust Implementation
-[](https://crates.io/crates/parquet)
+[](https://crates.io/crates/parquet)
+[](https://docs.rs/parquet/latest/parquet/)
This crate contains the official Native Rust implementation of [Apache Parquet](https://parquet.apache.org/), which is part of the [Apache Arrow](https://arrow.apache.org/) project.
See [crate documentation](https://docs.rs/parquet/latest/parquet/) for examples and the full API.
-## Rust Version Compatbility
+## Rust Version Compatibility
This crate is tested with the latest stable version of Rust. We do not currently test against other, older versions of the Rust compiler.
-## Features
+## Versioning / Releases
+
+The arrow crate follows the [SemVer standard](https://doc.rust-lang.org/cargo/reference/semver.html) defined by Cargo and works well within the Rust crate ecosystem.
+
+However, for historical reasons, this crate uses versions with major numbers greater than `0.x` (e.g. `19.0.0`), unlike many other crates in the Rust ecosystem which spend extended time releasing versions `0.x` to signal planned ongoing API changes. Minor arrow releases contain only compatible changes, while major releases may contain breaking API changes.
+
+## Feature Flags
+
+The `parquet` crate provides the following features which may be enabled in your `Cargo.toml`:
+
+- `arrow` (default) - support for reading / writing [`arrow`](https://crates.io/crates/arrow) arrays to / from parquet
Review Comment:
here are the parquet feature flags I could find
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org