You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/11/04 20:35:26 UTC

[GitHub] [incubator-pinot] apucher opened a new pull request #6235: Data and query generator bytes-column support

apucher opened a new pull request #6235:
URL: https://github.com/apache/incubator-pinot/pull/6235


   ## Description
   Add support for raw BYTES columns to data and query generators to expand future testing capabilities. Furthermore, support multi-value columns in the data generator and increase control over dialect and query logging in the query runner.
   
   ## Upgrade Notes
   Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
   **No**
   
   Does this PR fix a zero-downtime upgrade introduced earlier?
   **No**
   
   Does this PR otherwise need attention when creating release notes? Things to consider:
   **No**
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6235: Data and query generator bytes-column support

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on a change in pull request #6235:
URL: https://github.com/apache/incubator-pinot/pull/6235#discussion_r521620768



##########
File path: pinot-tools/src/main/java/org/apache/pinot/tools/data/generator/DataGeneratorSpec.java
##########
@@ -41,6 +41,7 @@
 
   private final Map<String, DataType> dataTypesMap;
   private final Map<String, FieldType> fieldTypesMap;
+  private final Map<String, Boolean> singleValueFlagsMap;

Review comment:
       Are we capable of generating MV values?

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java
##########
@@ -182,7 +184,12 @@ public QueryGenerator(List<File> avroFiles, String pinotTableName, String h2Tabl
    * @param avroValue Avro value.
    */
   private static void storeAvroValueIntoValueSet(Set<String> valueSet, Object avroValue) {
-    if (avroValue instanceof Number) {
+    if (avroValue instanceof ByteBuffer) {
+      // for raw bytes
+      String hexRaw = StringUtils.stripStart(Hex.encodeHexString(((ByteBuffer) avroValue).array()), "0");
+      String hexAligned = (hexRaw.length() & 0x1) == 0 ? hexRaw : "0" + hexRaw;

Review comment:
       Why not directly using `Hex.encodeHexString(((ByteBuffer) avroValue).array())`?

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java
##########
@@ -182,7 +184,12 @@ public QueryGenerator(List<File> avroFiles, String pinotTableName, String h2Tabl
    * @param avroValue Avro value.
    */
   private static void storeAvroValueIntoValueSet(Set<String> valueSet, Object avroValue) {
-    if (avroValue instanceof Number) {
+    if (avroValue instanceof ByteBuffer) {
+      // for raw bytes
+      String hexRaw = StringUtils.stripStart(Hex.encodeHexString(((ByteBuffer) avroValue).array()), "0");
+      String hexAligned = (hexRaw.length() & 0x1) == 0 ? hexRaw : "0" + hexRaw;

Review comment:
       Also, does Avro always have `ByteBuffer` backed by an array?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io commented on pull request #6235: Data and query generator bytes-column support

Posted by GitBox <gi...@apache.org>.

codecov-io commented on pull request #6235:
URL: https://github.com/apache/incubator-pinot/pull/6235#issuecomment-721988398


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6235?src=pr&el=h1) Report
   > Merging [#6235](https://codecov.io/gh/apache/incubator-pinot/pull/6235?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **decrease** coverage by `20.07%`.
   > The diff coverage is `48.64%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6235/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6235?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #6235       +/-   ##
   ===========================================
   - Coverage   66.44%   46.37%   -20.08%     
   ===========================================
     Files        1075     1240      +165     
     Lines       54773    58940     +4167     
     Branches     8168     8737      +569     
   ===========================================
   - Hits        36396    27331     -9065     
   - Misses      15700    29367    +13667     
   + Partials     2677     2242      -435     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration | `46.37% <48.64%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6235?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
   | [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ache/pinot/broker/queryquota/QueryQuotaEntity.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9RdWVyeVF1b3RhRW50aXR5LmphdmE=) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [...routing/instanceselector/BaseInstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0Jhc2VJbnN0YW5jZVNlbGVjdG9yLmphdmE=) | `96.03% <ø> (-3.97%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `26.66% <0.00%> (-30.48%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `22.22% <0.00%> (-26.62%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `64.28% <ø> (-8.89%)` | :arrow_down: |
   | ... and [1229 more](https://codecov.io/gh/apache/incubator-pinot/pull/6235/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6235?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6235?src=pr&el=footer). Last update [0f4ee68...f87b940](https://codecov.io/gh/apache/incubator-pinot/pull/6235?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org