You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2021/06/03 12:35:34 UTC
[arrow-datafusion] branch master updated: use prettier check in CI
(#453)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/master by this push:
new a1b8305 use prettier check in CI (#453)
a1b8305 is described below
commit a1b8305cec4a9d2cd16797a876f98a5de410f605
Author: Jiayu Liu <Ji...@users.noreply.github.com>
AuthorDate: Thu Jun 3 20:33:39 2021 +0800
use prettier check in CI (#453)
* use prettier check in CI
* format two more files
* fix docs/specification/invariants.md
---
.github/workflows/dev.yml | 15 ++-
DEVELOPERS.md | 4 +-
docs/specification/invariants.md | 22 ++--
docs/specification/output-field-name-semantic.md | 132 +++++++++++------------
4 files changed, 88 insertions(+), 85 deletions(-)
diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml
index 6c6dd83..3f52ccd 100644
--- a/.github/workflows/dev.yml
+++ b/.github/workflows/dev.yml
@@ -23,7 +23,6 @@ on:
pull_request:
jobs:
-
lint:
name: Lint C++, Python, R, Rust, Docker, RAT
runs-on: ubuntu-latest
@@ -37,3 +36,17 @@ jobs:
run: pip install -e dev/archery[docker]
- name: Lint
run: archery lint --rat
+
+ prettier:
+ name: Use prettier to check formatting of documents
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v2
+ - uses: actions/setup-node@v2
+ with:
+ node-version: "14"
+ - name: Prettier check
+ run: |
+ # if you encounter error, try rerun the command below with --write instead of --check
+ # and commit the changes
+ npx prettier@2.3.0 --check {ballista,datafusion,datafusion-examples,dev,docs,python}/**/*.md README.md DEVELOPERS.md
diff --git a/DEVELOPERS.md b/DEVELOPERS.md
index b915e29..cd0792f 100644
--- a/DEVELOPERS.md
+++ b/DEVELOPERS.md
@@ -105,8 +105,8 @@ new specifications as you see fit.
Here is the list current active specifications:
-* [Output field name semantic](docs/specification/output-field-name-semantic.md)
-* [Invariants](docs/specification/invariants.md)
+- [Output field name semantic](docs/specification/output-field-name-semantic.md)
+- [Invariants](docs/specification/invariants.md)
## How to format `.md` document
diff --git a/docs/specification/invariants.md b/docs/specification/invariants.md
index 628bb77..17b7c1d 100644
--- a/docs/specification/invariants.md
+++ b/docs/specification/invariants.md
@@ -42,11 +42,11 @@ know what they need to enforce at the coding level.
## Notation
-* Field or physical field: the tuple name, `arrow::DataType` and nullability flag (a bool whether values can be null), represented in this document by `PF(name, type, nullable)`
-* Logical field: Field with a relation name. Represented in this document by `LF(relation, name, type, nullable)`
-* Projected plan: plan with projection as the root node.
-* Logical schema: a vector of logical fields, used by logical plan.
-* Physical schema: a vector of physical fields, used by both physical plan and Arrow record batch.
+- Field or physical field: the tuple name, `arrow::DataType` and nullability flag (a bool whether values can be null), represented in this document by `PF(name, type, nullable)`
+- Logical field: Field with a relation name. Represented in this document by `LF(relation, name, type, nullable)`
+- Projected plan: plan with projection as the root node.
+- Logical schema: a vector of logical fields, used by logical plan.
+- Physical schema: a vector of physical fields, used by both physical plan and Arrow record batch.
### Logical
@@ -62,8 +62,8 @@ logical_field(lf1: LF, lf2: LF, ...) -> LF
Examples:
-* `plus(a,b) -> LF(None, "{a} Plus {b}", d(a.type,b.type), a.nullable | b.nullable)` where d is the function mapping input types to output type (`get_supertype` in our current implementation).
-* `length(a) -> LF(None, "length({a})", u32, a.nullable)`
+- `plus(a,b) -> LF(None, "{a} Plus {b}", d(a.type,b.type), a.nullable | b.nullable)` where d is the function mapping input types to output type (`get_supertype` in our current implementation).
+- `length(a) -> LF(None, "length({a})", u32, a.nullable)`
#### Plan
@@ -91,8 +91,8 @@ physical_field(PF1, PF2, ...) -> PF
Examples:
-* `plus(a,b) -> PF("{a} Plus {b}", d(a.type,b.type), a.nullable | b.nullable)` where d is a complex function (`get_supertype` in our current implementation) whose computation is for each element in the columns, sum the two entries together and return it in the same type as the smallest type of both columns.
-* `length(&str) -> PF("length({a})", u32, a.nullable)` whose computation is "count number of bytes in the string".
+- `plus(a,b) -> PF("{a} Plus {b}", d(a.type,b.type), a.nullable | b.nullable)` where d is a complex function (`get_supertype` in our current implementation) whose computation is for each element in the columns, sum the two entries together and return it in the same type as the smallest type of both columns.
+- `length(&str) -> PF("length({a})", u32, a.nullable)` whose computation is "count number of bytes in the string".
#### Plan
@@ -216,8 +216,8 @@ guarantee this invariant.
In particular:
-* The derived DataType matches the code it uses to build the array for every branch of valid input type combinations.
-* The nullability flag matches how the values are built.
+- The derived DataType matches the code it uses to build the array for every branch of valid input type combinations.
+- The nullability flag matches how the values are built.
#### Validation
diff --git a/docs/specification/output-field-name-semantic.md b/docs/specification/output-field-name-semantic.md
index fd28d11..0407a17 100644
--- a/docs/specification/output-field-name-semantic.md
+++ b/docs/specification/output-field-name-semantic.md
@@ -23,20 +23,20 @@ This specification documents how field names in output record batches should be
generated based on given user queries. The filed name rules apply to
Datafusion queries planned from both SQL queries and Dataframe APIs.
-## Field name rules
-
-* All field names MUST not contain relation/table qualifier.
- * Both `SELECT t1.id`, `SELECT id` and `df.select_columns(&["id"])` SHOULD result in field name: `id`
-* Function names MUST be converted to lowercase.
- * `SELECT AVG(c1)` SHOULD result in field name: `avg(c1)`
-* Literal string MUST not be wrapped with quotes or double quotes.
- * `SELECT 'foo'` SHOULD result in field name: `foo`
-* Operator expressions MUST be wrapped with parentheses.
- * `SELECT -2` SHOULD result in field name: `(- 2)`
-* Operator and operand MUST be separated by spaces.
- * `SELECT 1+2` SHOULD result in field name: `(1 + 2)`
-* Function arguments MUST be separated by a comma `,` and a space.
- * `SELECT f(c1,c2)` and `df.select(vec![f.udf("f")?.call(vec![col("c1"), col("c2")])])` SHOULD result in field name: `f(c1, c2)`
+## Field name rules
+
+- All field names MUST not contain relation/table qualifier.
+ - Both `SELECT t1.id`, `SELECT id` and `df.select_columns(&["id"])` SHOULD result in field name: `id`
+- Function names MUST be converted to lowercase.
+ - `SELECT AVG(c1)` SHOULD result in field name: `avg(c1)`
+- Literal string MUST not be wrapped with quotes or double quotes.
+ - `SELECT 'foo'` SHOULD result in field name: `foo`
+- Operator expressions MUST be wrapped with parentheses.
+ - `SELECT -2` SHOULD result in field name: `(- 2)`
+- Operator and operand MUST be separated by spaces.
+ - `SELECT 1+2` SHOULD result in field name: `(1 + 2)`
+- Function arguments MUST be separated by a comma `,` and a space.
+ - `SELECT f(c1,c2)` and `df.select(vec![f.udf("f")?.call(vec![col("c1"), col("c2")])])` SHOULD result in field name: `f(c1, c2)`
## Appendices
@@ -66,26 +66,24 @@ JOIN t2 ON t1.id = t2.id
Datafusion Arrow record batches output:
-| id | a | id | b |
-|----|-----|----|-------|
-| 1 | foo | 1 | hello |
-| 2 | bar | 2 | world |
-
+| id | a | id | b |
+| --- | --- | --- | ----- |
+| 1 | foo | 1 | hello |
+| 2 | bar | 2 | world |
Spark, MySQL 8 and PostgreSQL 13 output:
-| id | a | id | b |
-|----|-----|----|-------|
-| 1 | foo | 1 | hello |
-| 2 | bar | 2 | world |
+| id | a | id | b |
+| --- | --- | --- | ----- |
+| 1 | foo | 1 | hello |
+| 2 | bar | 2 | world |
SQLite 3 output:
-| id | a | b |
-|----|-----|-------|
-| 1 | foo | hello |
-| 2 | bar | world |
-
+| id | a | b |
+| --- | --- | ----- |
+| 1 | foo | hello |
+| 2 | bar | world |
#### Function transformed columns
@@ -98,41 +96,38 @@ SELECT ABS(t1.id), abs(-id) FROM t1;
Datafusion Arrow record batches output:
| abs(id) | abs((- id)) |
-|---------|-------------|
+| ------- | ----------- |
| 1 | 1 |
| 2 | 2 |
-
Spark output:
| abs(id) | abs((- id)) |
-|---------|-------------|
+| ------- | ----------- |
| 1 | 1 |
| 2 | 2 |
-
MySQL 8 output:
| ABS(t1.id) | abs(-id) |
-|------------|----------|
+| ---------- | -------- |
| 1 | 1 |
| 2 | 2 |
PostgreSQL 13 output:
| abs | abs |
-|-----|-----|
+| --- | --- |
| 1 | 1 |
| 2 | 2 |
SQlite 3 output:
| ABS(t1.id) | abs(-id) |
-|------------|----------|
+| ---------- | -------- |
| 1 | 1 |
| 2 | 2 |
-
#### Function with operators
Query:
@@ -143,40 +138,38 @@ SELECT t1.id + ABS(id), ABS(id * t1.id) FROM t1;
Datafusion Arrow record batches output:
-| id + abs(id) | abs(id * id) |
-|--------------|--------------|
-| 2 | 1 |
-| 4 | 4 |
-
+| id + abs(id) | abs(id \* id) |
+| ------------ | ------------- |
+| 2 | 1 |
+| 4 | 4 |
Spark output:
-| id + abs(id) | abs(id * id) |
-|--------------|--------------|
-| 2 | 1 |
-| 4 | 4 |
+| id + abs(id) | abs(id \* id) |
+| ------------ | ------------- |
+| 2 | 1 |
+| 4 | 4 |
MySQL 8 output:
-| t1.id + ABS(id) | ABS(id * t1.id) |
-|-----------------|-----------------|
-| 2 | 1 |
-| 4 | 4 |
+| t1.id + ABS(id) | ABS(id \* t1.id) |
+| --------------- | ---------------- |
+| 2 | 1 |
+| 4 | 4 |
PostgreSQL output:
| ?column? | abs |
-|----------|-----|
+| -------- | --- |
| 2 | 1 |
| 4 | 4 |
SQLite output:
-| t1.id + ABS(id) | ABS(id * t1.id) |
-|-----------------|-----------------|
-| 2 | 1 |
-| 4 | 4 |
-
+| t1.id + ABS(id) | ABS(id \* t1.id) |
+| --------------- | ---------------- |
+| 2 | 1 |
+| 4 | 4 |
#### Project literals
@@ -188,33 +181,30 @@ SELECT 1, 2+5, 'foo_bar';
Datafusion Arrow record batches output:
-| 1 | (2 + 5) | foo_bar |
-|---|---------|---------|
-| 1 | 7 | foo_bar |
-
+| 1 | (2 + 5) | foo_bar |
+| --- | ------- | ------- |
+| 1 | 7 | foo_bar |
Spark output:
-| 1 | (2 + 5) | foo_bar |
-|---|---------|---------|
-| 1 | 7 | foo_bar |
+| 1 | (2 + 5) | foo_bar |
+| --- | ------- | ------- |
+| 1 | 7 | foo_bar |
MySQL output:
-| 1 | 2+5 | foo_bar |
-|---|-----|---------|
-| 1 | 7 | foo_bar |
-
+| 1 | 2+5 | foo_bar |
+| --- | --- | ------- |
+| 1 | 7 | foo_bar |
PostgreSQL output:
| ?column? | ?column? | ?column? |
-|----------|----------|----------|
+| -------- | -------- | -------- |
| 1 | 7 | foo_bar |
-
SQLite 3 output:
-| 1 | 2+5 | 'foo_bar' |
-|---|-----|-----------|
-| 1 | 7 | foo_bar |
+| 1 | 2+5 | 'foo_bar' |
+| --- | --- | --------- |
+| 1 | 7 | foo_bar |