You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ege-st (via GitHub)" <gi...@apache.org> on 2024/02/23 15:30:44 UTC
[PR] Refactor: Remove unused schema validator code [pinot]
ege-st opened a new pull request, #12481:
URL: https://github.com/apache/pinot/pull/12481
The `IngestionSchemaValidator` logic was not used in any code path except for 3 unit tests. It appears as if it was used to validate the schema of a segment versus an Avro schema (as Avro was the only implementation of the Schema Validator). But the validator itself is never executed in any code path of Pinot.
This also removes the `getColumnStatisticsCollector` from `SegmentIndexCreationDriver` as that method was never being called anywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor: Remove unused schema validator code [pinot]
Posted by "ege-st (via GitHub)" <gi...@apache.org>.
ege-st commented on PR #12481:
URL: https://github.com/apache/pinot/pull/12481#issuecomment-1962560953
No issues, but since I couldn't find any path within core Pinot that was
using this logic, I thought the code which uses it might have been removed
at some point.
On Sat, Feb 24, 2024, 1:42 PM Xiaotian (Jackie) Jiang <
***@***.***> wrote:
> @ege-st <https://github.com/ege-st> They are part of pinot-spi which is
> intended to be used by some external tool (not maintained in this repo). Do
> you run into issues with them?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/pinot/pull/12481#issuecomment-1962511791>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BAASDJZUIINJY2QDVSRCEE3YVIYALAVCNFSM6AAAAABDW5F3ZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRSGUYTCNZZGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor: Remove unused schema validator code [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed pull request #12481: Refactor: Remove unused schema validator code
URL: https://github.com/apache/pinot/pull/12481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor: Remove unused schema validator code [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #12481:
URL: https://github.com/apache/pinot/pull/12481#issuecomment-1962511791
@ege-st They are part of `pinot-spi` which is intended to be used by some external tool (not maintained in this repo). Do you run into issues with them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor: Remove unused schema validator code [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12481:
URL: https://github.com/apache/pinot/pull/12481#issuecomment-1961622501
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12481?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 61.73%. Comparing base [(`fe75309`)](https://app.codecov.io/gh/apache/pinot/commit/fe7530969dc850fc46e8346732d2695e73bdd09a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`1780496`)](https://app.codecov.io/gh/apache/pinot/pull/12481?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12481 +/- ##
=========================================
Coverage 61.72% 61.73%
Complexity 207 207
=========================================
Files 2436 2434 -2
Lines 133219 133127 -92
Branches 20635 20619 -16
=========================================
- Hits 82233 82189 -44
+ Misses 44940 44896 -44
+ Partials 6046 6042 -4
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (+<0.01%)` | :arrow_up: |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (+<0.01%)` | :arrow_up: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.69% <ø> (+0.01%)` | :arrow_up: |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.87% <ø> (-26.74%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.71% <ø> (+0.01%)` | :arrow_up: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.85% <ø> (-26.74%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.73% <ø> (+<0.01%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.73% <ø> (+<0.01%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.88% <ø> (+0.02%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12481/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.72% <ø> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12481?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor: Remove unused schema validator code [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #12481:
URL: https://github.com/apache/pinot/pull/12481#issuecomment-1962058350
@jackjlli I saw you contributed this. Is this still in use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor: Remove unused schema validator code [pinot]
Posted by "ege-st (via GitHub)" <gi...@apache.org>.
ege-st commented on PR #12481:
URL: https://github.com/apache/pinot/pull/12481#issuecomment-1962547282
No worries. We can kill this pr.
On Fri, Feb 23, 2024, 6:49 PM Jialiang Li ***@***.***> wrote:
> ***@***.**** requested changes on this pull request.
>
> Hi @ege-st <https://github.com/ege-st>, this schema validator is actively
> being used inside LinkedIn. Basically it attempts to detect any schema
> mismatch during segment generation. The stats would be useful to know the
> quality of the input data. So please do not remove this part of code.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/pinot/pull/12481#pullrequestreview-1899099705>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BAASDJ2OCAF3LX75WHSUVILYVETHNAVCNFSM6AAAAABDW5F3ZGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQOJZGA4TSNZQGU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org