You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Dracylfrr (via GitHub)" <gi...@apache.org> on 2026/06/13 23:02:12 UTC

[I] Add Column Summary Statistics workflow operator [texera]

Dracylfrr opened a new issue, #5693:
URL: https://github.com/apache/texera/issues/5693

   ### Feature Summary
   
   Texera workflows often involve exploring a dataset before applying cleaning, visualization, or analysis operators. A basic column-level summary operator would help users quickly understand the shape and quality of an input table.
   
   This issue proposes adding a Column Summary Statistics workflow operator that takes one input table and outputs one summary row per input column.
   
   Initial output fields:
   
   * columnName
   * dataType
   * rowCount
   * nullCount
   * nonNullCount
   * minValue
   * maxValue
   * meanValue
   
   
   ### Proposed Solution or Design
   
   For the first version:
   
   * Numeric columns should report min, max, and mean.
   * Non-numeric columns should report row/null/non-null counts and leave numeric summary fields null.
   * The operator should follow existing Texera native operator patterns.
   * Unit tests should cover numeric columns, non-numeric columns, null values, mixed columns, and empty input.
   
   This is intended as a focused workflow operator for basic per-column summary statistics.
   
   ### Affected Area
   
   Workflow Engine (Amber)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #5693:
URL: https://github.com/apache/texera/issues/5693#issuecomment-4793978350

   sounds good. Let's close this issue and the linked PR as not planned. and you can create a new issue and a new PR for `Describe` operator. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed issue #5693: Add Column Summary Statistics workflow operator
URL: https://github.com/apache/texera/issues/5693


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "Dracylfrr (via GitHub)" <gi...@apache.org>.
Dracylfrr commented on issue #5693:
URL: https://github.com/apache/texera/issues/5693#issuecomment-4764503397

   Hi @Yicong-Huang,
   
   Thanks for pointing this out. I see the concern about duplicating the existing result panel table statistics.
   
   The distinction I had in mind is that the current result panel statistics are primarily for interactively viewing statistics after an operator emits a table, while this proposed operator would expose those statistics as an actual workflow output table. That would allow the summary statistics to be consumed by downstream operators, exported, saved, compared across workflow runs, or used as part of a repeatable data-profiling / cleaning workflow.
   
   For example, a user could run this operator after loading a dataset and then feed the generated summary table into later workflow steps instead of only inspecting the statistics manually in the UI.
   
   If this workflow-output use case is not considered useful enough for Texera, I understand that this may duplicate the existing result panel feature. In that case, I’d be happy to redirect the work toward improving or extending the existing table statistics feature instead if needed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #5693:
URL: https://github.com/apache/texera/issues/5693#issuecomment-4785334583

   Hi @Dracylfrr after consideration I do think this "column summary statistic" operator is a bit overkill, for two reasons:
   1. we do have statistics feature on result table for display quick summary of the columns.
   2. to get those statistics as a dataset (for downstream operators to use) we want to direct users to use `Aggregate` operator which is a more standard building block. The column summary operator sounds like a syntax sugar, or a preconfigured aggregation. 
   
   However, I see this operator can be convenient for users. A better name or a reposition could be introduce a `Describe` operator, so it is not just showing aggregated result of a column, but also to describe a table. The description can up to your definition. WDYT? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on issue #5693:
URL: https://github.com/apache/texera/issues/5693#issuecomment-4765112835

   @Dracylfrr If possible, can you include two figures (screenshots or hand drawings) to show the differences?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #5693:
URL: https://github.com/apache/texera/issues/5693#issuecomment-4762982450

   Thanks, @Dracylfrr. We already have statistics in the result panel table view, which works for all operators that emit tables.
   
   Do we need a separate operator for this use case? I’m wondering whether that would duplicate the existing table statistics feature, unless there is some additional functionality or workflow-specific need that the result panel does not cover.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "Dracylfrr (via GitHub)" <gi...@apache.org>.
Dracylfrr commented on issue #5693:
URL: https://github.com/apache/texera/issues/5693#issuecomment-4793886505

   Hi @chenlica and @Yicong-Huang, thank you for the feedback. That makes sense to me.
   
   I understand the concern that the current **Column Summary Statistics** operator may be too close to existing functionality:
   
   1. The result table statistics feature already provides a quick summary for display and inspection.
   2. The Aggregate operator is the standard workflow building block when users want to manually compute statistics as an output dataset.
   
   I agree that if this operator is only viewed as a preconfigured aggregation, then it becomes more like syntax sugar over Aggregate. I think repositioning it as a **Describe** operator would make the purpose clearer.
   
   The way I see it, the **Describe** operator would focus on describing the input table as a whole, similar to `df.describe()`, `df.shape`, or SQL `DESCRIBE`, instead of only acting as a shortcut for aggregation.
   
   Here are two rough figures to show the difference I had in mind.
   
   **Figure 1: Existing approaches**
   
   ```text
   Input Table
      |
      |------------------------------|
      |                              |
      v                              v
   Result Table Statistics        Aggregate Operator
   (UI quick summary)             (user-configured aggregation)
      |                              |
      v                              v
   Displayed in result panel       Output dataset for downstream operators
   ```
   
   In the existing approach, result table statistics are useful for quick UI inspection, while the Aggregate operator is useful when users know which aggregations they want to configure.
   
   **Figure 2: Proposed Describe operator**
   
   ```text
   Input Table
      |
      v
   Describe Operator
      |
      v
   Table Description Dataset
   
   Example output:
   
   columnName | dataType | rowCount | nullCount | nonNullCount | minValue | maxValue | meanValue
   age        | integer  | 1000     | 20        | 980          | 18       | 85       | 42.5
   name       | string   | 1000     | 5         | 995          | -        | -        | -
   createdAt  | datetime | 1000     | 0         | 1000         | -        | -        | -
   ```
   
   The intended difference is that **Describe** would generate a reusable table/profile description as workflow output. This could then be connected to downstream operators, exported, or reused in a repeatable workflow. So the focus would be less on “preconfigured aggregation” and more on “describe this table.”
   
   I can update the PR direction toward this **Describe** operator framing if this matches what you had in mind.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add Column Summary Statistics workflow operator [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5693:
URL: https://github.com/apache/texera/issues/5693#issuecomment-4700051400

   <!-- texera:welcome-first-time-contributor -->
   👋 Thanks for your first contribution to Texera, @Dracylfrr!
   
   If you're looking for a good place to start, browse issues labeled [`starter-task`](https://github.com/apache/texera/issues?q=is%3Aissue+is%3Aopen+label%3Astarter-task); they're scoped to be approachable for newcomers.
   
   You can drive common housekeeping yourself by commenting one of these commands on its own line:
   
   - **Issues.** Comment `/take` to assign an open issue to yourself, or `/untake` to release it. You can find unclaimed work with the search filter `is:issue is:open no:assignee`.
   - **Sub-issues.** To link issues into a parent/child hierarchy, comment `/sub-issue #5166 #5222` on the parent to attach those children (or `/unsub-issue #5166 #5222` to detach them). From a child issue, comment `/parent-issue #5166` to set its parent, or `/unparent-issue` to clear it (the current parent is detected automatically). References may be written as `#5166` or as a bare `5166`; cross-repository references are not supported.
   - **Pull requests (author only).** Comment `/request-review @user` to request a review from someone, or `/unrequest-review @user` to withdraw that request.
   
   Each command must match exactly: `/take this` will not work, only `/take` does. For the full contribution flow, see [CONTRIBUTING.md](https://github.com/apache/texera/blob/main/CONTRIBUTING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org