You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2022/09/01 19:56:48 UTC
[pinot] branch release-0.11-rc updated (956888b093 -> 24b92eaf0b)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch release-0.11-rc
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard 956888b093 [maven-release-plugin] prepare for next development iteration
omit d40e1cd18b [maven-release-plugin] prepare release release-0.11.0-rc0
omit a7379a51c9 include contrib directory as it's required for build source code
omit 584a782afb Separate assembly files to generate different targets
omit aaebcefe6c Add capabilities to ingest from another stream without disabling the realtime table (#9289) (#9316)
omit 58e2740acd Force commit consuming segments (#9197) (#9315)
omit a6b264cbf8 update NOTICE-binary
omit 0efae04367 more license files
omit 5c7e436225 Update license
omit 0e048f5328 Sync pom dependencies for jdk8 related libs
new b4a8b8f540 Required changes from 0.11.0 release (#9314)
new 2e10747002 Force commit consuming segments (#9197)
new ce49c4f4cc Add capabilities to ingest from another stream without disabling the realtime table (#9289)
new 24b92eaf0b Include fmpp plugin module inside the src as it's required for building source code (#9321)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (956888b093)
\
N -- N -- N refs/heads/release-0.11-rc (24b92eaf0b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
contrib/pinot-fmpp-maven-plugin/pom.xml | 6 +-
pinot-broker/pom.xml | 5 +-
pinot-clients/pinot-java-client/pom.xml | 5 +-
pinot-clients/pinot-jdbc-client/pom.xml | 8 +-
pinot-clients/pom.xml | 6 +-
pinot-common/pom.xml | 43 ++--
pinot-compatibility-verifier/pom.xml | 5 +-
pinot-connectors/pinot-flink-connector/pom.xml | 8 +-
pinot-connectors/pinot-spark-connector/pom.xml | 5 +-
pinot-connectors/pom.xml | 6 +-
.../pinot-common-jdk8/pom.xml | 7 +-
.../pinot-core-jdk8/pom.xml | 5 +-
.../pinot-java-client-jdk8/pom.xml | 5 +-
.../pinot-segment-local-jdk8/pom.xml | 5 +-
.../pinot-segment-spi-jdk8/pom.xml | 5 +-
.../pinot-spi-jdk8/pom.xml | 5 +-
.../prestodb-pinot-dependencies/pom.xml | 6 +-
.../presto-pinot-driver/pom.xml | 2 +-
pinot-controller/pom.xml | 5 +-
pinot-core/pom.xml | 5 +-
pinot-distribution/pinot-assembly.xml | 1 +
pinot-distribution/pinot-tarball-assembly.xml | 233 ---------------------
pinot-distribution/pom.xml | 45 +++-
pinot-integration-test-base/pom.xml | 5 +-
pinot-integration-tests/pom.xml | 5 +-
pinot-minion/pom.xml | 5 +-
pinot-perf/pom.xml | 5 +-
.../pinot-batch-ingestion-common/pom.xml | 6 +-
.../pinot-batch-ingestion-hadoop/pom.xml | 6 +-
.../pinot-batch-ingestion-spark-2.4/pom.xml | 6 +-
.../pinot-batch-ingestion-spark-3.2/pom.xml | 6 +-
.../pinot-batch-ingestion-standalone/pom.xml | 6 +-
pinot-plugins/pinot-batch-ingestion/pom.xml | 6 +-
.../v0_deprecated/pinot-hadoop/pom.xml | 7 +-
.../v0_deprecated/pinot-ingestion-common/pom.xml | 6 +-
.../v0_deprecated/pinot-spark/pom.xml | 7 +-
.../pinot-batch-ingestion/v0_deprecated/pom.xml | 6 +-
.../pinot-environment/pinot-azure/pom.xml | 6 +-
pinot-plugins/pinot-environment/pom.xml | 6 +-
pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 +-
pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 +-
pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 5 +-
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 +-
pinot-plugins/pinot-file-system/pom.xml | 6 +-
.../pinot-input-format/pinot-avro-base/pom.xml | 5 +-
.../pinot-input-format/pinot-avro/pom.xml | 5 +-
.../pinot-confluent-avro/pom.xml | 5 +-
pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 +-
.../pinot-input-format/pinot-json/pom.xml | 5 +-
pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 6 +-
.../pinot-input-format/pinot-parquet/pom.xml | 5 +-
.../pinot-input-format/pinot-protobuf/pom.xml | 5 +-
.../pinot-input-format/pinot-thrift/pom.xml | 5 +-
pinot-plugins/pinot-input-format/pom.xml | 6 +-
.../pinot-metrics/pinot-dropwizard/pom.xml | 5 +-
pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 +-
pinot-plugins/pinot-metrics/pom.xml | 6 +-
.../pinot-minion-builtin-tasks/pom.xml | 5 +-
pinot-plugins/pinot-minion-tasks/pom.xml | 6 +-
.../pinot-segment-uploader-default/pom.xml | 6 +-
pinot-plugins/pinot-segment-uploader/pom.xml | 6 +-
.../pinot-segment-writer-file-based/pom.xml | 6 +-
pinot-plugins/pinot-segment-writer/pom.xml | 6 +-
.../pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 9 +-
.../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 +-
.../pinot-kafka-base/pom.xml | 6 +-
.../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 +-
.../pinot-stream-ingestion/pinot-pulsar/pom.xml | 6 +-
pinot-plugins/pinot-stream-ingestion/pom.xml | 6 +-
pinot-plugins/pom.xml | 11 +-
pinot-query-planner/pom.xml | 5 +-
pinot-query-runtime/pom.xml | 5 +-
pinot-segment-local/pom.xml | 5 +-
pinot-segment-spi/pom.xml | 5 +-
pinot-server/pom.xml | 5 +-
pinot-spi/pom.xml | 5 +-
pinot-tools/pom.xml | 5 +-
pom.xml | 11 +-
78 files changed, 330 insertions(+), 426 deletions(-)
delete mode 100644 pinot-distribution/pinot-tarball-assembly.xml
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[pinot] 01/04: Required changes from 0.11.0 release (#9314)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.11-rc
in repository https://gitbox.apache.org/repos/asf/pinot.git
commit b4a8b8f540f9af7ac5ecc494f5bd99e99614b586
Author: Xiang Fu <xi...@gmail.com>
AuthorDate: Wed Aug 31 20:24:33 2022 -0700
Required changes from 0.11.0 release (#9314)
* Sync pom dependencies for jdk8 related libs
* Update license
* more license files
* update NOTICE-binary
---
LICENSE-binary | 247 ++++++++++++---------
NOTICE-binary | 173 ++++-----------
licenses-binary/LICENSE-grizzled-slf4j.txt | 36 +++
licenses-binary/LICENSE-janino.txt | 31 +++
licenses-binary/LICENSE-jbcrypt.txt | 17 ++
licenses-binary/LICENSE-kryo.txt | 10 +
licenses-binary/LICENSE-minlog.txt | 10 +
licenses-binary/LICENSE-scopt.txt | 21 ++
.../pinot-common-jdk8/pom.xml | 10 +
.../pinot-core-jdk8/pom.xml | 20 ++
.../pinot-java-client-jdk8/pom.xml | 3 +-
11 files changed, 342 insertions(+), 236 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 5367a6e438..a8fd7eb074 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -204,27 +204,28 @@
This project bundles some components that are also licensed under the Apache
License Version 2.0:
+cloud.localstack:localstack-utils:0.2.19
com.101tec:zkclient:0.7
com.chuusai:shapeless_2.12:2.3.3
com.clearspring.analytics:stream:2.7.0
-com.fasterxml.jackson.core:jackson-annotations:2.10.0
-com.fasterxml.jackson.core:jackson-core:2.10.0
-com.fasterxml.jackson.core:jackson-databind:2.10.0
-com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.10.4
-com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.10.5
-com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.10.0
-com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.0
-com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.5
-com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.3
-com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.10.0
-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.10.0
-com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.10.0
-com.fasterxml.jackson.module:jackson-module-jsonSchema:2.11.1
-com.fasterxml.jackson.module:jackson-module-paranamer:2.10.0
-com.fasterxml.jackson.module:jackson-module-scala_2.11:2.6.7.1
-com.fasterxml.jackson.module:jackson-module-scala_2.12:2.10.0
-com.fasterxml.woodstox:woodstox-core:6.0.1
+com.esri.geometry:esri-geometry-api:2.2.0
+com.fasterxml.jackson.core:jackson-annotations:2.12.7
+com.fasterxml.jackson.core:jackson-core:2.12.7
+com.fasterxml.jackson.core:jackson-databind:2.12.7
+com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.7
+com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.12.7
+com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.12.7
+com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.12.7
+com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.12.7
+com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.12.7
+com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.12.7
+com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.12.7
+com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.12.7
+com.fasterxml.jackson.module:jackson-module-jsonSchema:2.12.7
+com.fasterxml.jackson.module:jackson-module-scala_2.12:2.12.7
+com.fasterxml.woodstox:woodstox-core:5.0.3
com.github.ben-manes.caffeine:caffeine:2.6.2
+com.github.os72:protobuf-dynamic:1.0.1
com.google.android:annotations:4.1.1.4
com.google.api-client:google-api-client:1.30.10
com.google.api.grpc:proto-google-common-protos:1.18.1
@@ -249,14 +250,22 @@ com.google.oauth-client:google-oauth-client:1.31.0
com.google.uzaygezen:uzaygezen-core:0.2
com.jayway.jsonpath:json-path:2.7.0
com.lmax:disruptor:3.3.4
-com.ning:compress-lzf:1.0.3
-com.rabbitmq:amqp-client:5.12.0
com.squareup.okio:okio:1.6.0
+com.squareup.wire:wire-runtime:3.2.2
+com.squareup.wire:wire-schema:3.2.2
com.tdunning:t-digest:3.2
+com.twitter:chill-java:0.7.6
+com.twitter:chill_2.12:0.7.6
+com.typesafe.akka:akka-actor_2.12:2.5.21
+com.typesafe.akka:akka-protobuf_2.12:2.5.21
+com.typesafe.akka:akka-slf4j_2.12:2.5.21
+com.typesafe.akka:akka-stream_2.12:2.5.21
com.typesafe.netty:netty-reactive-streams-http:2.0.4
com.typesafe.netty:netty-reactive-streams:2.0.4
com.typesafe.scala-logging:scala-logging_2.12:3.9.2
-com.uber:h3:3.0.3
+com.typesafe:config:1.3.3
+com.typesafe:ssl-config-core_2.12:0.3.7
+com.uber:h3:3.7.2
com.yahoo.datasketches:memory:0.8.3
com.yahoo.datasketches:sketches-core:0.8.3
com.yammer.metrics:metrics-core:2.2.0
@@ -269,7 +278,6 @@ commons-httpclient:commons-httpclient:3.1
commons-io:commons-io:2.11.0
commons-lang:commons-lang:2.6
commons-logging:commons-logging:1.2
-commons-net:commons-net:3.1
commons-pool:commons-pool:1.6
info.picocli:picocli:4.6.1
io.airlift:aircompressor:0.10
@@ -278,62 +286,78 @@ io.circe:circe-generic_2.12:0.13.0
io.circe:circe-jawn_2.12:0.13.0
io.circe:circe-numbers_2.12:0.13.0
io.circe:circe-parser_2.12:0.13.0
-io.confluent:common-config:5.3.1
-io.confluent:common-utils:5.3.1
-io.confluent:kafka-avro-serializer:5.3.1
-io.confluent:kafka-schema-registry-client:5.3.1
+io.confluent:common-config:5.5.3
+io.confluent:common-utils:5.5.3
+io.confluent:kafka-avro-serializer:5.5.3
+io.confluent:kafka-protobuf-serializer:5.5.3
+io.confluent:kafka-schema-registry-client:5.5.3
+io.confluent:kafka-schema-serializer:5.5.3
io.dropwizard.metrics:metrics-core:4.2.2
-io.dropwizard.metrics:metrics-graphite:4.2.2
io.dropwizard.metrics:metrics-jmx:4.2.2
-io.dropwizard.metrics:metrics-json:4.2.2
-io.dropwizard.metrics:metrics-jvm:4.2.2
io.grpc:grpc-api:1.41.0
io.grpc:grpc-context:1.14.0
io.grpc:grpc-core:1.41.0
io.grpc:grpc-netty-shaded:1.41.0
io.grpc:grpc-protobuf-lite:1.19.0
-io.grpc:grpc-protobuf:1.30.0
-io.grpc:grpc-stub:1.30.0
-io.netty:netty-all:4.1.74.Final
-io.netty:netty-buffer:4.1.74.Final
-io.netty:netty-codec-dns:4.1.74.Final
-io.netty:netty-codec-haproxy:4.1.74.Final
-io.netty:netty-codec-http2:4.1.74.Final
-io.netty:netty-codec-http:4.1.74.Final
-io.netty:netty-codec-socks:4.1.74.Final
-io.netty:netty-codec:4.1.74.Final
-io.netty:netty-common:4.1.74.Final
-io.netty:netty-handler-proxy:4.1.74.Final
-io.netty:netty-handler:4.1.74.Final
-io.netty:netty-resolver-dns:4.1.74.Final
-io.netty:netty-resolver:4.1.74.Final
-io.netty:netty-tcnative-boringssl-static:2.0.36.Final
-io.netty:netty-transport-native-epoll:4.1.74.Final
-io.netty:netty-transport-native-kqueue:4.1.74.Final
-io.netty:netty-transport-native-unix-common:4.1.74.Final
-io.netty:netty-transport:4.1.74.Final
-io.netty:netty:3.9.6.Final
+io.grpc:grpc-protobuf:1.41.0
+io.grpc:grpc-stub:1.41.0
+io.netty:netty-all:4.1.79.Final
+io.netty:netty-buffer:4.1.79.Final
+io.netty:netty-codec-dns:4.1.79.Final
+io.netty:netty-codec-haproxy:4.1.79.Final
+io.netty:netty-codec-http2:4.1.79.Final
+io.netty:netty-codec-http:4.1.79.Final
+io.netty:netty-codec-memcache:4.1.79.Final
+io.netty:netty-codec-mqtt:4.1.79.Final
+io.netty:netty-codec-redis:4.1.79.Final
+io.netty:netty-codec-smtp:4.1.79.Final
+io.netty:netty-codec-socks:4.1.79.Final
+io.netty:netty-codec-stomp:4.1.79.Final
+io.netty:netty-codec-xml:4.1.79.Final
+io.netty:netty-codec:4.1.79.Final
+io.netty:netty-common:4.1.79.Final
+io.netty:netty-handler-proxy:4.1.79.Final
+io.netty:netty-handler:4.1.79.Final
+io.netty:netty-resolver-dns-classes-macos:4.1.79.Final
+io.netty:netty-resolver-dns-native-macos:4.1.79.Final
+io.netty:netty-resolver-dns-native-macos:4.1.79.Final
+io.netty:netty-resolver-dns:4.1.79.Final
+io.netty:netty-resolver:4.1.79.Final
+io.netty:netty-tcnative-boringssl-static:2.0.53.Final
+io.netty:netty-tcnative-classes:2.0.53.Final
+io.netty:netty-transport-classes-epoll:4.1.79.Final
+io.netty:netty-transport-classes-kqueue:4.1.79.Final
+io.netty:netty-transport-native-epoll:4.1.79.Final
+io.netty:netty-transport-native-epoll:4.1.79.Final
+io.netty:netty-transport-native-kqueue:4.1.79.Final
+io.netty:netty-transport-native-kqueue:4.1.79.Final
+io.netty:netty-transport-native-unix-common:4.1.79.Final
+io.netty:netty-transport-rxtx:4.1.79.Final
+io.netty:netty-transport-sctp:4.1.79.Final
+io.netty:netty-transport-udt:4.1.79.Final
+io.netty:netty-transport:4.1.79.Final
io.opencensus:opencensus-api:0.24.0
io.opencensus:opencensus-contrib-http-util:0.24.0
io.perfmark:perfmark-api:0.23.0
-io.projectreactor.netty:reactor-netty:0.9.15.RELEASE
-io.projectreactor:reactor-core:3.3.12.RELEASE
+io.projectreactor.netty:reactor-netty-core:1.0.15
+io.projectreactor.netty:reactor-netty-http:1.0.15
+io.projectreactor:reactor-core:3.4.14
io.prometheus:simpleclient:0.8.1
io.prometheus:simpleclient_common:0.8.1
io.prometheus:simpleclient_hotspot:0.8.1
-io.swagger:swagger-annotations:1.5.21
-io.swagger:swagger-core:1.5.16
-io.swagger:swagger-jaxrs:1.5.16
-io.swagger:swagger-jersey2-jaxrs:1.5.16
-io.swagger:swagger-models:1.5.16
+io.swagger:swagger-annotations:1.6.2
+io.swagger:swagger-core:1.6.6
+io.swagger:swagger-jaxrs:1.6.6
+io.swagger:swagger-jersey2-jaxrs:1.6.6
+io.swagger:swagger-models:1.6.6
it.unimi.dsi:fastutil:8.2.3
+jakarta.validation:jakarta.validation-api:2.0.2
javax.inject:javax.inject:1
javax.validation:validation-api:2.0.1.Final
joda-time:joda-time:2.10.5
-net.minidev:accessors-smart:2.4.7
-net.minidev:json-smart:2.4.7
-org.apache.avro:avro-ipc:1.8.2
-org.apache.avro:avro-mapred:1.8.2
+net.java.dev.jna:jna:4.1.0
+net.minidev:accessors-smart:1.2
+net.minidev:json-smart:2.3
org.apache.avro:avro-protobuf:1.9.1
org.apache.avro:avro:1.9.2
org.apache.bookkeeper:bookkeeper-common-allocator:4.12.0
@@ -344,19 +368,36 @@ org.apache.calcite:calcite-core:1.29.0
org.apache.calcite:calcite-linq4j:1.29.0
org.apache.commons:commons-collections4:4.1
org.apache.commons:commons-compress:1.21
-org.apache.commons:commons-crypto:1.0.0
org.apache.commons:commons-csv:1.0
-org.apache.commons:commons-lang3:3.5
+org.apache.commons:commons-lang3:3.11
org.apache.commons:commons-math3:3.2
org.apache.commons:commons-math:2.1
org.apache.datasketches:datasketches-java:1.2.0-incubating
org.apache.datasketches:datasketches-memory:1.2.0-incubating
-org.apache.helix:helix-core:0.9.8
+org.apache.flink:flink-annotations:1.12.0
+org.apache.flink:flink-core:1.12.0
+org.apache.flink:flink-file-sink-common:1.12.0
+org.apache.flink:flink-hadoop-fs:1.12.0
+org.apache.flink:flink-java:1.12.0
+org.apache.flink:flink-metrics-core:1.12.0
+org.apache.flink:flink-queryable-state-client-java:1.12.0
+org.apache.flink:flink-runtime_2.12:1.12.0
+org.apache.flink:flink-shaded-asm-7:7.1-12.0
+org.apache.flink:flink-shaded-guava:18.0-12.0
+org.apache.flink:flink-shaded-jackson:2.10.1-12.0
+org.apache.flink:flink-shaded-netty:4.1.49.Final-12.0
+org.apache.flink:flink-shaded-zookeeper-3:3.4.14-12.0
+org.apache.flink:flink-streaming-java_2.12:1.12.0
+org.apache.flink:force-shading:1.12.0
+org.apache.helix:helix-common:1.0.4
+org.apache.helix:helix-core:1.0.4
+org.apache.helix:metadata-store-directory-common:1.0.4
+org.apache.helix:metrics-common:1.0.4
+org.apache.helix:zookeeper-api:1.0.4
org.apache.hive:hive-storage-api:2.7.1
org.apache.httpcomponents:httpclient:4.5.13
org.apache.httpcomponents:httpcore:4.4.13
org.apache.httpcomponents:httpmime:4.5.13
-org.apache.ivy:ivy:2.4.0
org.apache.kafka:kafka-clients:2.8.1
org.apache.kafka:kafka-metadata:2.8.1
org.apache.kafka:kafka-raft:2.8.1
@@ -387,15 +428,7 @@ org.apache.pulsar:pulsar-client-api:2.7.2
org.apache.pulsar:pulsar-client-original:2.7.2
org.apache.pulsar:pulsar-common:2.7.2
org.apache.pulsar:pulsar-transaction-common:2.7.2
-org.apache.spark:spark-core_2.11:2.4.0
-org.apache.spark:spark-kvstore_2.11:2.4.0
-org.apache.spark:spark-launcher_2.11:2.4.0
-org.apache.spark:spark-network-common_2.11:2.4.0
-org.apache.spark:spark-network-shuffle_2.11:2.4.0
-org.apache.spark:spark-tags_2.11:2.4.0
-org.apache.spark:spark-unsafe_2.11:2.4.0
org.apache.thrift:libthrift:0.15.0
-org.apache.xbean:xbean-asm6-shaded:4.8
org.apache.yetus:audience-annotations:0.13.0
org.apache.zookeeper:zookeeper-jute:3.5.8
org.apache.zookeeper:zookeeper:3.5.8
@@ -413,26 +446,25 @@ org.eclipse.jetty:jetty-servlet:9.4.45.v20220203
org.eclipse.jetty:jetty-util-ajax:9.4.45.v20220203
org.eclipse.jetty:jetty-util:9.4.45.v20220203
org.javassist:javassist:3.19.0-GA
-org.json4s:json4s-ast_2.11:3.5.3
-org.json4s:json4s-core_2.11:3.5.3
-org.json4s:json4s-jackson_2.11:3.5.3
-org.json4s:json4s-scalap_2.11:3.5.3
+org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.3.71
+org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.3.71
+org.jetbrains.kotlin:kotlin-stdlib:1.4.0
+org.jetbrains:annotations:13.0
org.lz4:lz4-java:1.8.0
+org.objenesis:objenesis:2.1
org.quartz-scheduler:quartz:2.3.2
org.roaringbitmap:RoaringBitmap:0.9.28
org.roaringbitmap:shims:0.9.28
org.scala-lang.modules:scala-collection-compat_2.12:2.3.0
org.scala-lang.modules:scala-java8-compat_2.12:0.9.1
org.scala-lang.modules:scala-xml_2.12:1.3.0
-org.spark-project.spark:unused:1.0.0
org.typelevel:macro-compat_2.12:1.1.1
org.webjars:swagger-ui:3.23.11
org.wildfly.openssl:wildfly-openssl:1.0.7.Final
org.xerial.larray:larray-buffer:0.4.1
org.xerial.larray:larray-mmap:0.4.1
-org.xerial.snappy:snappy-java:1.1.1.7
+org.xerial.snappy:snappy-java:1.1.8.2
org.yaml:snakeyaml:1.30
-oro:oro:2.0.8
software.amazon.awssdk:annotations:2.14.28
software.amazon.awssdk:apache-client:2.14.28
software.amazon.awssdk:arns:2.14.28
@@ -454,6 +486,7 @@ software.amazon.awssdk:sdk-core:2.14.28
software.amazon.awssdk:sts:2.14.28
software.amazon.awssdk:utils:2.14.28
software.amazon.eventstream:eventstream:1.0.1
+xml-apis:xml-apis:1.4.01
------------------------------------------------------------------------------------
@@ -464,14 +497,14 @@ of these licenses.
MIT License
-----------
-com.azure:azure-core-http-netty:1.7.1
-com.azure:azure-core:1.12.0
-com.azure:azure-storage-blob:12.10.0
-com.azure:azure-storage-common:12.10.0
-com.azure:azure-storage-file-datalake:12.4.0
-com.azure:azure-storage-internal-avro:12.0.2
+com.azure:azure-core-http-netty:1.11.8
+com.azure:azure-core:1.26.0
+com.azure:azure-storage-blob:12.15.0
+com.azure:azure-storage-common:12.15.0
+com.azure:azure-storage-file-datalake:12.8.0
+com.azure:azure-storage-internal-avro:12.2.0
com.microsoft.azure:azure-data-lake-store-sdk:2.3.9
-net.razorvine:pyrolite:4.13
+com.github.scopt:scopt_2.12:3.5.0
net.sf.jopt-simple:jopt-simple:4.6
net.sourceforge.argparse4j:argparse4j:0.7.0
org.checkerframework:checker-compat-qual:2.5.5
@@ -482,8 +515,7 @@ org.typelevel:cats-core_2.12:2.1.0
org.typelevel:cats-kernel_2.12:2.1.0
org.typelevel:cats-macros_2.12:2.1.0
org.typelevel:jawn-parser_2.12:1.0.0
-org.slf4j:jcl-over-slf4j:1.7.16
-org.slf4j:jul-to-slf4j:1.7.16
+org.slf4j:jul-to-slf4j:1.7.25
org.slf4j:slf4j-api:1.7.25
@@ -494,28 +526,31 @@ com.thoughtworks.paranamer:paranamer:2.8
BSD 2-Clause
------------
-com.github.luben:zstd-jni:1.4.9-5
-net.sf.py4j:py4j:0.10.7
-org.codehaus.woodstox:stax2-api:4.2
+com.github.luben:zstd-jni:1.5.2-3
+org.clapper:grizzled-slf4j_2.12:1.3.2
+org.codehaus.woodstox:stax2-api:3.1.4
BSD 3-Clause
------------
+com.esotericsoftware.kryo:kryo:2.24.0
+com.esotericsoftware.minlog:minlog:1.2
com.google.api:api-common:1.10.0
com.google.api:gax-httpjson:0.75.2
com.google.api:gax:1.58.2
com.google.auth:google-auth-library-credentials:0.21.1
com.google.auth:google-auth-library-oauth2-http:0.21.1
com.google.protobuf:protobuf-java-util:3.13.0
-com.google.protobuf:protobuf-java:3.12.0
+com.google.protobuf:protobuf-java:3.19.2
com.google.protobuf:protobuf-lite:3.0.1
com.jcabi:jcabi-aspects:0.22
com.jcabi:jcabi-log:0.17.1
org.antlr:antlr4-runtime:4.6
-org.fusesource.leveldbjni:leveldbjni-all:1.8
-org.ow2.asm:asm:9.1
-org.scala-lang:scala-library:2.11.11
-org.scala-lang:scala-reflect:2.11.8
+org.codehaus.janino:commons-compiler:3.1.6
+org.codehaus.janino:janino:3.1.6
+org.ow2.asm:asm:5.0.4
+org.scala-lang:scala-library:2.10.5
+org.scala-lang.modules:scala-parser-combinators_2.12:1.1.1
org.scala-lang.modules:scala-xml_2.11:1.0.6
org.threeten:threeten-extra:1.5.0
org.threeten:threetenbp:1.4.4
@@ -525,10 +560,10 @@ Common Development and Distribution License (CDDL) 1.0
------------------------------------------------------
(see licenses/LICENSE-cddl-1.0.txt)
-javax.activation:activation:1.1.1
-javax.annotation:javax.annotation-api:1.3.2
+javax.activation:activation:1.2.0
+javax.annotation:javax.annotation-api:1.2
javax.servlet:javax.servlet-api:3.0.1
-org.glassfish.jersey.containers:jersey-container-servlet-core:2.25.1
+org.glassfish.jersey.containers:jersey-container-servlet-core:2.35
Common Development and Distribution License (CDDL) 1.1
@@ -536,9 +571,8 @@ Common Development and Distribution License (CDDL) 1.1
(see licenses/LICENSE-cddl-1.1.txt)
com.sun.xml.bind:jaxb-core:2.3.0
-com.sun.xml.bind:jaxb-impl:2.3.0
javax.ws.rs:javax.ws.rs-api:2.0.1
-javax.xml.bind:jaxb-api:2.3.0
+javax.xml.bind:jaxb-api:2.2.11
Eclipse Public License (EPL) 1.0
@@ -572,7 +606,6 @@ org.glassfish.jersey.core:jersey-common:2.35
org.glassfish.jersey.core:jersey-server:2.35
org.glassfish.jersey.ext:jersey-entity-filtering:2.35
org.glassfish.jersey.inject:jersey-hk2:2.35
-org.glassfish.jersey.media:jersey-media-jaxb:2.35
org.glassfish.jersey.media:jersey-media-json-jackson:2.35
org.glassfish.jersey.media:jersey-media-multipart:2.35
org.glassfish.tyrus.bundles:tyrus-standalone-client:1.15
@@ -585,12 +618,12 @@ Eclipse Distribution License (EDL) 1.0
com.sun.activation:jakarta.activation:1.2.2
jakarta.activation:jakarta.activation-api:1.2.1
jakarta.xml.bind:jakarta.xml.bind-api:2.3.2
-org.jvnet.mimepull:mimepull:1.9.11
+org.jvnet.mimepull:mimepull:1.9.13
WTFPL License
-------------
-org.reflections:reflections:0.9.9
+org.reflections:reflections:0.9.11
Creative Commons Attribution License (CC BY 2.5)
@@ -603,3 +636,9 @@ Bounty Castle License
org.bouncycastle:bcpkix-jdk15to18:1.68
org.bouncycastle:bcprov-ext-jdk15to18:1.68
org.bouncycastle:bcprov-jdk15to18:1.68
+
+
+ISC License
+-----------
+
+org.mindrot:jbcrypt:0.4
diff --git a/NOTICE-binary b/NOTICE-binary
index ab66d60bfc..814da7074b 100644
--- a/NOTICE-binary
+++ b/NOTICE-binary
@@ -49,22 +49,19 @@ Apache Commons Logging
Copyright 2003-2014 The Apache Software Foundation
Apache Commons Lang
-Copyright 2001-2016 The Apache Software Foundation
-
-This product includes software from the Spring Framework,
-under the Apache License 2.0 (see: StringUtils.containsWhitespace())
+Copyright 2001-2020 The Apache Software Foundation
Apache Commons Collections
Copyright 2001-2015 The Apache Software Foundation
Apache Log4j SLF4J Binding
-Copyright 1999-2019 The Apache Software Foundation
+Copyright 1999-1969 The Apache Software Foundation
Apache Log4j API
-Copyright 1999-2019 The Apache Software Foundation
+Copyright 1999-1969 The Apache Software Foundation
Apache Log4j 1.x Compatibility API
-Copyright 1999-2019 The Apache Software Foundation
+Copyright 1999-1969 The Apache Software Foundation
=============================================================================
= NOTICE file corresponding to section 4d of the Apache License Version 2.0 =
@@ -77,19 +74,16 @@ Joda.org (https://www.joda.org/).
Jackson is a high-performance, Free/Open Source JSON processing library.
It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
been in development since 2007.
-It is currently developed by a community of developers, as well as supported
-commercially by FasterXML.com.
+It is currently developed by a community of developers.
## Licensing
-Jackson core and extension components may be licensed under different licenses.
+Jackson 2.x core and extension components are licensed under Apache License 2.0
To find the details that apply to this artifact see the accompanying LICENSE file.
-For more information, including possible other licensing options, contact
-FasterXML.com (http://fasterxml.com).
## Credits
-A list of contributors may be found from CREDITS file, which is included
+A list of contributors may be found from CREDITS(-2.x) file, which is included
in some artifacts (usually source distributions); but is always available
from the source code management (SCM) system project uses.
@@ -129,6 +123,21 @@ The web site includes files generated by Jekyll.
Apache Calcite -- Avatica
Copyright 2012-2021 The Apache Software Foundation
+Jackson is a high-performance, Free/Open Source JSON processing library.
+It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+been in development since 2007.
+It is currently developed by a community of developers, as well as supported
+commercially by FasterXML.com.
+
+Jackson core and extension components may be licensed under different licenses.
+To find the details that apply to this artifact see the accompanying LICENSE file.
+For more information, including possible other licensing options, contact
+FasterXML.com (http://fasterxml.com).
+
+A list of contributors may be found from CREDITS file, which is included
+in some artifacts (usually source distributions); but is always available
+from the source code management (SCM) system project uses.
+
Apache Thrift
Copyright (C) 2006 - 2019, The Apache Software Foundation
@@ -187,30 +196,40 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Julian Seward, jseward@acm.org
Apache Helix :: Core
-Copyright 2020 The Apache Software Foundation
+Copyright 2022 The Apache Software Foundation
+
+Apache Helix :: Helix Common
+Copyright 2022 The Apache Software Foundation
-This product currently only contains code developed by authors
-of specific components, as identified by the source code files;
-if such notes are missing files have been created by
-Tatu Saloranta.
+Apache Helix :: Metrics Common
+Copyright 2022 The Apache Software Foundation
-For additional credits (generally to people who reported problems)
-see CREDITS file.
+Apache Helix :: ZooKeeper API
+Copyright 2022 The Apache Software Foundation
- The Netty Project
+Apache Helix :: Metadata Store Directory Common
+Copyright 2022 The Apache Software Foundation
+
+Apache ZooKeeper - Server
+Copyright 2008-2020 The Apache Software Foundation
+
+Apache ZooKeeper - Jute
+Copyright 2008-2020 The Apache Software Foundation
+
+ The Netty Project
=================
Please visit the Netty web site for more information:
* http://netty.io/
-Copyright 2011 The Netty Project
+Copyright 2016 The Netty Project
The Netty Project licenses this file to you under the Apache License,
version 2.0 (the "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at:
-http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
@@ -218,114 +237,6 @@ WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
-Also, please refer to each LICENSE.<component>.txt file, which is located in
-the 'license' directory of the distribution file, for the license terms of the
-components that this product depends on.
-
--------------------------------------------------------------------------------
-This product contains the extensions to Java Collections Framework which has
-been derived from the works by JSR-166 EG, Doug Lea, and Jason T. Greene:
-
- * LICENSE:
- * license/LICENSE.jsr166y.txt (Public Domain)
- * HOMEPAGE:
- * http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/
- * http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbosscache/experimental/jsr166/
-
-This product contains a modified version of Robert Harder's Public Domain
-Base64 Encoder and Decoder, which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.base64.txt (Public Domain)
- * HOMEPAGE:
- * http://iharder.sourceforge.net/current/java/base64/
-
-This product contains a modified version of 'JZlib', a re-implementation of
-zlib in pure Java, which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.jzlib.txt (BSD Style License)
- * HOMEPAGE:
- * http://www.jcraft.com/jzlib/
-
-This product contains a modified version of 'Webbit', a Java event based
-WebSocket and HTTP server:
-
- * LICENSE:
- * license/LICENSE.webbit.txt (BSD License)
- * HOMEPAGE:
- * https://github.com/joewalnes/webbit
-
-This product optionally depends on 'Protocol Buffers', Google's data
-interchange format, which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.protobuf.txt (New BSD License)
- * HOMEPAGE:
- * http://code.google.com/p/protobuf/
-
-This product optionally depends on 'Bouncy Castle Crypto APIs' to generate
-a temporary self-signed X.509 certificate when the JVM does not provide the
-equivalent functionality. It can be obtained at:
-
- * LICENSE:
- * license/LICENSE.bouncycastle.txt (MIT License)
- * HOMEPAGE:
- * http://www.bouncycastle.org/
-
-This product optionally depends on 'SLF4J', a simple logging facade for Java,
-which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.slf4j.txt (MIT License)
- * HOMEPAGE:
- * http://www.slf4j.org/
-
-This product optionally depends on 'Apache Commons Logging', a logging
-framework, which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.commons-logging.txt (Apache License 2.0)
- * HOMEPAGE:
- * http://commons.apache.org/logging/
-
-This product optionally depends on 'Apache Log4J', a logging framework,
-which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.log4j.txt (Apache License 2.0)
- * HOMEPAGE:
- * http://logging.apache.org/log4j/
-
-This product optionally depends on 'JBoss Logging', a logging framework,
-which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.jboss-logging.txt (GNU LGPL 2.1)
- * HOMEPAGE:
- * http://anonsvn.jboss.org/repos/common/common-logging-spi/
-
-This product optionally depends on 'Apache Felix', an open source OSGi
-framework implementation, which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.felix.txt (Apache License 2.0)
- * HOMEPAGE:
- * http://felix.apache.org/
-
-Apache ZooKeeper - Server
-Copyright 2008-2020 The Apache Software Foundation
-
-Apache ZooKeeper - Jute
-Copyright 2008-2020 The Apache Software Foundation
-
- The Netty Project
- =================
-
-Copyright 2016 The Netty Project
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-------------------------------------------------------------------------------
This product contains a forked and modified version of Tomcat Native
diff --git a/licenses-binary/LICENSE-grizzled-slf4j.txt b/licenses-binary/LICENSE-grizzled-slf4j.txt
new file mode 100644
index 0000000000..9a1caa635c
--- /dev/null
+++ b/licenses-binary/LICENSE-grizzled-slf4j.txt
@@ -0,0 +1,36 @@
+ClassUtil Library License
+=========================
+
+This software is released under a BSD 3-clause license. See
+https://opensource.org/licenses/BSD-3-Clause
+
+---
+
+Copyright © 2010-2016, Brian M. Clapper.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors
+ may be used to endorse or promote products derived from this software without
+ specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGE.
\ No newline at end of file
diff --git a/licenses-binary/LICENSE-janino.txt b/licenses-binary/LICENSE-janino.txt
new file mode 100644
index 0000000000..d1e1f237c4
--- /dev/null
+++ b/licenses-binary/LICENSE-janino.txt
@@ -0,0 +1,31 @@
+Janino - An embedded Java[TM] compiler
+
+Copyright (c) 2001-2016, Arno Unkrig
+Copyright (c) 2015-2016 TIBCO Software Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+ 1. Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials
+ provided with the distribution.
+ 3. Neither the name of JANINO nor the names of its contributors
+ may be used to endorse or promote products derived from this
+ software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
+OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
+IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\ No newline at end of file
diff --git a/licenses-binary/LICENSE-jbcrypt.txt b/licenses-binary/LICENSE-jbcrypt.txt
new file mode 100644
index 0000000000..d332534c06
--- /dev/null
+++ b/licenses-binary/LICENSE-jbcrypt.txt
@@ -0,0 +1,17 @@
+jBCrypt is subject to the following license:
+
+/*
+ * Copyright (c) 2006 Damien Miller <dj...@mindrot.org>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
diff --git a/licenses-binary/LICENSE-kryo.txt b/licenses-binary/LICENSE-kryo.txt
new file mode 100644
index 0000000000..bf531009ca
--- /dev/null
+++ b/licenses-binary/LICENSE-kryo.txt
@@ -0,0 +1,10 @@
+Copyright (c) 2008-2022, Nathan Sweet
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+* Neither the name of Esoteric Software nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROF [...]
\ No newline at end of file
diff --git a/licenses-binary/LICENSE-minlog.txt b/licenses-binary/LICENSE-minlog.txt
new file mode 100644
index 0000000000..3f6a160c23
--- /dev/null
+++ b/licenses-binary/LICENSE-minlog.txt
@@ -0,0 +1,10 @@
+Copyright (c) 2008, Nathan Sweet
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+ * Neither the name of Esoteric Software nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROF [...]
\ No newline at end of file
diff --git a/licenses-binary/LICENSE-scopt.txt b/licenses-binary/LICENSE-scopt.txt
new file mode 100644
index 0000000000..180d282df2
--- /dev/null
+++ b/licenses-binary/LICENSE-scopt.txt
@@ -0,0 +1,21 @@
+This project is licensed under the [MIT license](https://en.wikipedia.org/wiki/MIT_License).
+
+Copyright (c) scopt contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
+OR OTHER DEALINGS IN THE SOFTWARE.
\ No newline at end of file
diff --git a/pinot-connectors/prestodb-pinot-dependencies/pinot-common-jdk8/pom.xml b/pinot-connectors/prestodb-pinot-dependencies/pinot-common-jdk8/pom.xml
index 6286434ef6..2c7a301829 100644
--- a/pinot-connectors/prestodb-pinot-dependencies/pinot-common-jdk8/pom.xml
+++ b/pinot-connectors/prestodb-pinot-dependencies/pinot-common-jdk8/pom.xml
@@ -314,6 +314,10 @@
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>javax.xml.bind</groupId>
+ <artifactId>jaxb-api</artifactId>
+ </exclusion>
</exclusions>
</dependency>
<dependency>
@@ -406,6 +410,12 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
diff --git a/pinot-connectors/prestodb-pinot-dependencies/pinot-core-jdk8/pom.xml b/pinot-connectors/prestodb-pinot-dependencies/pinot-core-jdk8/pom.xml
index d04c17a9d1..f0698d474f 100644
--- a/pinot-connectors/prestodb-pinot-dependencies/pinot-core-jdk8/pom.xml
+++ b/pinot-connectors/prestodb-pinot-dependencies/pinot-core-jdk8/pom.xml
@@ -110,6 +110,26 @@
</exclusion>
</exclusions>
</dependency>-->
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
+ <classifier>linux-x86_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ <classifier>osx-x86_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-tcnative-boringssl-static</artifactId>
+ <classifier>linux-x86_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-tcnative-boringssl-static</artifactId>
+ <classifier>osx-x86_64</classifier>
+ </dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
diff --git a/pinot-connectors/prestodb-pinot-dependencies/pinot-java-client-jdk8/pom.xml b/pinot-connectors/prestodb-pinot-dependencies/pinot-java-client-jdk8/pom.xml
index 87d04f03ff..8d0faca62b 100644
--- a/pinot-connectors/prestodb-pinot-dependencies/pinot-java-client-jdk8/pom.xml
+++ b/pinot-connectors/prestodb-pinot-dependencies/pinot-java-client-jdk8/pom.xml
@@ -56,7 +56,8 @@
<dependencies>
<dependency>
<groupId>org.apache.pinot</groupId>
- <artifactId>pinot-spi</artifactId>
+ <artifactId>pinot-spi-jdk8</artifactId>
+ <version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[pinot] 02/04: Force commit consuming segments (#9197)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.11-rc
in repository https://gitbox.apache.org/repos/asf/pinot.git
commit 2e10747002479eb889b124485e1feece1ca519a1
Author: Sajjad Moradi <mo...@gmail.com>
AuthorDate: Mon Aug 29 11:43:30 2022 -0700
Force commit consuming segments (#9197)
---
.../api/resources/PinotRealtimeTableResource.java | 20 ++++++++++++++++++++
.../realtime/PinotLLCRealtimeSegmentManager.java | 9 +++++++++
2 files changed, 29 insertions(+)
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java
index ba03a6cc5c..7bf4ac4235 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java
@@ -90,6 +90,26 @@ public class PinotRealtimeTableResource {
}
}
+ @POST
+ @Path("/tables/{tableName}/forceCommit")
+ @ApiOperation(value = "Force commit the current consuming segments",
+ notes = "Force commit the current segments in consuming state and restart consumption. "
+ + "This should be used after schema/table config changes. "
+ + "Please note that this is an asynchronous operation, "
+ + "and 200 response does not mean it has actually been done already")
+ public Response forceCommit(
+ @ApiParam(value = "Name of the table", required = true) @PathParam("tableName") String tableName) {
+ String tableNameWithType = TableNameBuilder.REALTIME.tableNameWithType(tableName);
+ validate(tableNameWithType);
+ try {
+ _pinotLLCRealtimeSegmentManager.forceCommit(tableNameWithType);
+ return Response.ok().build();
+ } catch (Exception e) {
+ throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.INTERNAL_SERVER_ERROR, e);
+ }
+ }
+
+
@GET
@Path("/tables/{tableName}/pauseStatus")
@Produces(MediaType.APPLICATION_JSON)
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
index fa4f3063e0..5cec6b8c4b 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
@@ -1420,6 +1420,15 @@ public class PinotLLCRealtimeSegmentManager {
}
}
+ /**
+ * Force commit the current segments in consuming state and restart consumption
+ */
+ public void forceCommit(String tableNameWithType) {
+ IdealState idealState = getIdealState(tableNameWithType);
+ Set<String> consumingSegments = findConsumingSegments(idealState);
+ sendForceCommitMessageToServers(tableNameWithType, consumingSegments);
+ }
+
/**
* Pause consumption on a table by
* 1) setting "isTablePaused" in ideal states to true and
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[pinot] 03/04: Add capabilities to ingest from another stream without disabling the realtime table (#9289)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.11-rc
in repository https://gitbox.apache.org/repos/asf/pinot.git
commit ce49c4f4cc293ae9629d36d59ef83cda2240c543
Author: Sajjad Moradi <mo...@gmail.com>
AuthorDate: Mon Aug 29 17:44:51 2022 -0700
Add capabilities to ingest from another stream without disabling the realtime table (#9289)
---
.../api/resources/PinotRealtimeTableResource.java | 20 ++--
.../realtime/PinotLLCRealtimeSegmentManager.java | 113 +++++++++++++--------
.../RealtimeSegmentValidationManager.java | 10 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 5 +-
4 files changed, 98 insertions(+), 50 deletions(-)
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java
index 7bf4ac4235..715d65bede 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java
@@ -31,6 +31,7 @@ import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
+import javax.ws.rs.QueryParam;
import javax.ws.rs.core.HttpHeaders;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
@@ -61,8 +62,7 @@ public class PinotRealtimeTableResource {
@POST
@Path("/tables/{tableName}/pauseConsumption")
@Produces(MediaType.APPLICATION_JSON)
- @ApiOperation(value = "Pause consumption of a realtime table",
- notes = "Pause the consumption of a realtime table")
+ @ApiOperation(value = "Pause consumption of a realtime table", notes = "Pause the consumption of a realtime table")
public Response pauseConsumption(
@ApiParam(value = "Name of the table", required = true) @PathParam("tableName") String tableName) {
String tableNameWithType = TableNameBuilder.REALTIME.tableNameWithType(tableName);
@@ -77,14 +77,22 @@ public class PinotRealtimeTableResource {
@POST
@Path("/tables/{tableName}/resumeConsumption")
@Produces(MediaType.APPLICATION_JSON)
- @ApiOperation(value = "Resume consumption of a realtime table",
- notes = "Resume the consumption for a realtime table")
+ @ApiOperation(value = "Resume consumption of a realtime table", notes =
+ "Resume the consumption for a realtime table. ConsumeFrom parameter indicates from which offsets "
+ + "consumption should resume. If consumeFrom parameter is not provided, consumption continues based on the "
+ + "offsets in segment ZK metadata, and in case the offsets are already gone, the first available offsets are "
+ + "picked to minimize the data loss.")
public Response resumeConsumption(
- @ApiParam(value = "Name of the table", required = true) @PathParam("tableName") String tableName) {
+ @ApiParam(value = "Name of the table", required = true) @PathParam("tableName") String tableName,
+ @ApiParam(value = "smallest | largest") @QueryParam("consumeFrom") String consumeFrom) {
String tableNameWithType = TableNameBuilder.REALTIME.tableNameWithType(tableName);
validate(tableNameWithType);
+ if (consumeFrom != null && !consumeFrom.equalsIgnoreCase("smallest") && !consumeFrom.equalsIgnoreCase("largest")) {
+ throw new ControllerApplicationException(LOGGER,
+ String.format("consumeFrom param '%s' is not valid.", consumeFrom), Response.Status.BAD_REQUEST);
+ }
try {
- return Response.ok(_pinotLLCRealtimeSegmentManager.resumeConsumption(tableNameWithType)).build();
+ return Response.ok(_pinotLLCRealtimeSegmentManager.resumeConsumption(tableNameWithType, consumeFrom)).build();
} catch (Exception e) {
throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.INTERNAL_SERVER_ERROR, e);
}
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
index 5cec6b8c4b..762ffdc421 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
@@ -873,7 +873,7 @@ public class PinotLLCRealtimeSegmentManager {
* which means it's manually triggered by admin not by automatic periodic task)
*/
public void ensureAllPartitionsConsuming(TableConfig tableConfig, PartitionLevelStreamConfig streamConfig,
- boolean recreateDeletedConsumingSegment) {
+ boolean recreateDeletedConsumingSegment, OffsetCriteria offsetCriteria) {
Preconditions.checkState(!_isStopping, "Segment manager is stopping");
String realtimeTableName = tableConfig.getTableName();
@@ -881,17 +881,20 @@ public class PinotLLCRealtimeSegmentManager {
assert idealState != null;
boolean isTableEnabled = idealState.isEnabled();
boolean isTablePaused = isTablePaused(idealState);
+ boolean offsetsHaveToChange = offsetCriteria != null;
if (isTableEnabled && !isTablePaused) {
List<PartitionGroupConsumptionStatus> currentPartitionGroupConsumptionStatusList =
- getPartitionGroupConsumptionStatusList(idealState, streamConfig);
- // Read the smallest offset when a new partition is detected
+ offsetsHaveToChange
+ ? Collections.emptyList() // offsets from metadata are not valid anymore; fetch for all partitions
+ : getPartitionGroupConsumptionStatusList(idealState, streamConfig);
OffsetCriteria originalOffsetCriteria = streamConfig.getOffsetCriteria();
- streamConfig.setOffsetCriteria(OffsetCriteria.SMALLEST_OFFSET_CRITERIA);
+ // Read the smallest offset when a new partition is detected
+ streamConfig.setOffsetCriteria(offsetsHaveToChange ? offsetCriteria : OffsetCriteria.SMALLEST_OFFSET_CRITERIA);
List<PartitionGroupMetadata> newPartitionGroupMetadataList =
getNewPartitionGroupMetadataList(streamConfig, currentPartitionGroupConsumptionStatusList);
streamConfig.setOffsetCriteria(originalOffsetCriteria);
return ensureAllPartitionsConsuming(tableConfig, streamConfig, idealState, newPartitionGroupMetadataList,
- recreateDeletedConsumingSegment);
+ recreateDeletedConsumingSegment, offsetCriteria);
} else {
LOGGER.info("Skipping LLC segments validation for table: {}, isTableEnabled: {}, isTablePaused: {}",
realtimeTableName, isTableEnabled, isTablePaused);
@@ -1052,7 +1055,7 @@ public class PinotLLCRealtimeSegmentManager {
@VisibleForTesting
IdealState ensureAllPartitionsConsuming(TableConfig tableConfig, PartitionLevelStreamConfig streamConfig,
IdealState idealState, List<PartitionGroupMetadata> newPartitionGroupMetadataList,
- boolean recreateDeletedConsumingSegment) {
+ boolean recreateDeletedConsumingSegment, OffsetCriteria offsetCriteria) {
String realtimeTableName = tableConfig.getTableName();
InstancePartitions instancePartitions = getConsumingInstancePartitions(tableConfig);
@@ -1074,6 +1077,16 @@ public class PinotLLCRealtimeSegmentManager {
// Get the latest segment ZK metadata for each partition
Map<Integer, SegmentZKMetadata> latestSegmentZKMetadataMap = getLatestSegmentZKMetadataMap(realtimeTableName);
+ // create a map of <parition id, start offset> using data already fetched in newPartitionGroupMetadataList
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToStartOffset = new HashMap<>();
+ for (PartitionGroupMetadata metadata : newPartitionGroupMetadataList) {
+ partitionGroupIdToStartOffset.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ }
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToSmallestStreamOffset = null;
+ if (offsetCriteria == OffsetCriteria.SMALLEST_OFFSET_CRITERIA) {
+ partitionGroupIdToSmallestStreamOffset = partitionGroupIdToStartOffset;
+ }
+
// Walk over all partitions that we have metadata for, and repair any partitions necessary.
// Possible things to repair:
// 1. The latest metadata is in DONE state, but the idealstate says segment is CONSUMING:
@@ -1144,10 +1157,18 @@ public class PinotLLCRealtimeSegmentManager {
// 3. we should never end up with some replicas ONLINE and some OFFLINE.
if (isAllInstancesInState(instanceStateMap, SegmentStateModel.OFFLINE)) {
LOGGER.info("Repairing segment: {} which is OFFLINE for all instances in IdealState", latestSegmentName);
- StreamPartitionMsgOffset startOffset = offsetFactory.create(latestSegmentZKMetadata.getStartOffset());
+ if (partitionGroupIdToSmallestStreamOffset == null) {
+ // Smallest offset is fetched from stream once and stored in partitionGroupIdToSmallestStreamOffset.
+ // This is to prevent fetching the same info for each and every partition group.
+ partitionGroupIdToSmallestStreamOffset = fetchPartitionGroupIdToSmallestOffset(streamConfig);
+ }
+ StreamPartitionMsgOffset startOffset =
+ selectStartOffset(offsetCriteria, partitionGroupId, partitionGroupIdToStartOffset,
+ partitionGroupIdToSmallestStreamOffset, tableConfig.getTableName(), offsetFactory,
+ latestSegmentZKMetadata.getStartOffset()); // segments are OFFLINE; start from beginning
createNewConsumingSegment(tableConfig, streamConfig, latestSegmentZKMetadata, currentTimeMs,
- partitionGroupId, newPartitionGroupMetadataList, instancePartitions, instanceStatesMap,
- segmentAssignment, instancePartitionsMap, startOffset);
+ newPartitionGroupMetadataList, instancePartitions, instanceStatesMap, segmentAssignment,
+ instancePartitionsMap, startOffset);
} else {
if (newPartitionGroupSet.contains(partitionGroupId)) {
if (recreateDeletedConsumingSegment && latestSegmentZKMetadata.getStatus().isCompleted()
@@ -1155,10 +1176,18 @@ public class PinotLLCRealtimeSegmentManager {
// If we get here, that means in IdealState, the latest segment has all replicas ONLINE.
// Create a new IN_PROGRESS segment in PROPERTYSTORE,
// add it as CONSUMING segment to IDEALSTATE.
- StreamPartitionMsgOffset startOffset = offsetFactory.create(latestSegmentZKMetadata.getEndOffset());
+ if (partitionGroupIdToSmallestStreamOffset == null) {
+ // Smallest offset is fetched from stream once and stored in partitionGroupIdToSmallestStreamOffset.
+ // This is to prevent fetching the same info for each and every partition group.
+ partitionGroupIdToSmallestStreamOffset = fetchPartitionGroupIdToSmallestOffset(streamConfig);
+ }
+ StreamPartitionMsgOffset startOffset =
+ selectStartOffset(offsetCriteria, partitionGroupId, partitionGroupIdToStartOffset,
+ partitionGroupIdToSmallestStreamOffset, tableConfig.getTableName(), offsetFactory,
+ latestSegmentZKMetadata.getEndOffset());
createNewConsumingSegment(tableConfig, streamConfig, latestSegmentZKMetadata, currentTimeMs,
- partitionGroupId, newPartitionGroupMetadataList, instancePartitions, instanceStatesMap,
- segmentAssignment, instancePartitionsMap, startOffset);
+ newPartitionGroupMetadataList, instancePartitions, instanceStatesMap, segmentAssignment,
+ instancePartitionsMap, startOffset);
} else {
LOGGER.error("Got unexpected instance state map: {} for segment: {}", instanceStateMap,
latestSegmentName);
@@ -1220,7 +1249,7 @@ public class PinotLLCRealtimeSegmentManager {
}
private void createNewConsumingSegment(TableConfig tableConfig, PartitionLevelStreamConfig streamConfig,
- SegmentZKMetadata latestSegmentZKMetadata, long currentTimeMs, int partitionGroupId,
+ SegmentZKMetadata latestSegmentZKMetadata, long currentTimeMs,
List<PartitionGroupMetadata> newPartitionGroupMetadataList, InstancePartitions instancePartitions,
Map<String, Map<String, String>> instanceStatesMap, SegmentAssignment segmentAssignment,
Map<InstancePartitionsType, InstancePartitions> instancePartitionsMap, StreamPartitionMsgOffset startOffset) {
@@ -1228,23 +1257,6 @@ public class PinotLLCRealtimeSegmentManager {
int numPartitions = newPartitionGroupMetadataList.size();
LLCSegmentName latestLLCSegmentName = new LLCSegmentName(latestSegmentZKMetadata.getSegmentName());
LLCSegmentName newLLCSegmentName = getNextLLCSegmentName(latestLLCSegmentName, currentTimeMs);
- StreamPartitionMsgOffset partitionGroupSmallestOffset =
- getPartitionGroupSmallestOffset(streamConfig, partitionGroupId);
-
- if (partitionGroupSmallestOffset != null) {
- // Start offset must be higher than the start offset of the stream
- if (partitionGroupSmallestOffset.compareTo(startOffset) > 0) {
- LOGGER.error("Data lost from offset: {} to: {} for partition: {} of table: {}", startOffset,
- partitionGroupSmallestOffset, partitionGroupId, tableConfig.getTableName());
- _controllerMetrics.addMeteredTableValue(tableConfig.getTableName(), ControllerMeter.LLC_STREAM_DATA_LOSS, 1L);
- startOffset = partitionGroupSmallestOffset;
- }
- } else {
- LOGGER.error("Smallest offset for partition: {} of table: {} not found. Using startOffset: {}", partitionGroupId,
- tableConfig.getTableName(), startOffset);
- _controllerMetrics.addMeteredTableValue(tableConfig.getTableName(), ControllerMeter.LLC_STREAM_DATA_LOSS, 1L);
- }
-
CommittingSegmentDescriptor committingSegmentDescriptor =
new CommittingSegmentDescriptor(latestSegmentZKMetadata.getSegmentName(), startOffset.toString(), 0);
createNewSegmentZKMetadata(tableConfig, streamConfig, newLLCSegmentName, currentTimeMs, committingSegmentDescriptor,
@@ -1254,21 +1266,39 @@ public class PinotLLCRealtimeSegmentManager {
instancePartitionsMap);
}
- @Nullable
- private StreamPartitionMsgOffset getPartitionGroupSmallestOffset(StreamConfig streamConfig, int partitionGroupId) {
+ private Map<Integer, StreamPartitionMsgOffset> fetchPartitionGroupIdToSmallestOffset(StreamConfig streamConfig) {
OffsetCriteria originalOffsetCriteria = streamConfig.getOffsetCriteria();
streamConfig.setOffsetCriteria(OffsetCriteria.SMALLEST_OFFSET_CRITERIA);
- List<PartitionGroupMetadata> smallestOffsetCriteriaPartitionGroupMetadata =
+ List<PartitionGroupMetadata> partitionGroupMetadataList =
getNewPartitionGroupMetadataList(streamConfig, Collections.emptyList());
streamConfig.setOffsetCriteria(originalOffsetCriteria);
- StreamPartitionMsgOffset partitionStartOffset = null;
- for (PartitionGroupMetadata info : smallestOffsetCriteriaPartitionGroupMetadata) {
- if (info.getPartitionGroupId() == partitionGroupId) {
- partitionStartOffset = info.getStartOffset();
- break;
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToSmallestOffset = new HashMap<>();
+ for (PartitionGroupMetadata metadata : partitionGroupMetadataList) {
+ partitionGroupIdToSmallestOffset.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ }
+ return partitionGroupIdToSmallestOffset;
+ }
+
+ private StreamPartitionMsgOffset selectStartOffset(OffsetCriteria offsetCriteria, int partitionGroupId,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToStartOffset,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToSmallestStreamOffset, String tableName,
+ StreamPartitionMsgOffsetFactory offsetFactory, String startOffsetInSegmentZkMetadataStr) {
+ if (offsetCriteria != null) {
+ // use the fetched offset according to offset criteria
+ return partitionGroupIdToStartOffset.get(partitionGroupId);
+ } else {
+ // use offset from segment ZK metadata
+ StreamPartitionMsgOffset startOffsetInSegmentZkMetadata = offsetFactory.create(startOffsetInSegmentZkMetadataStr);
+ StreamPartitionMsgOffset streamSmallestOffset = partitionGroupIdToSmallestStreamOffset.get(partitionGroupId);
+ // Start offset in ZK must be higher than the start offset of the stream
+ if (streamSmallestOffset.compareTo(startOffsetInSegmentZkMetadata) > 0) {
+ LOGGER.error("Data lost from offset: {} to: {} for partition: {} of table: {}", startOffsetInSegmentZkMetadata,
+ streamSmallestOffset, partitionGroupId, tableName);
+ _controllerMetrics.addMeteredTableValue(tableName, ControllerMeter.LLC_STREAM_DATA_LOSS, 1L);
+ return streamSmallestOffset;
}
+ return startOffsetInSegmentZkMetadata;
}
- return partitionStartOffset;
}
private LLCSegmentName getNextLLCSegmentName(LLCSegmentName lastLLCSegmentName, long creationTimeMs) {
@@ -1448,12 +1478,15 @@ public class PinotLLCRealtimeSegmentManager {
* 1) setting "isTablePaused" in ideal states to false and
* 2) triggering segment validation job to create new consuming segments in ideal states
*/
- public PauseStatus resumeConsumption(String tableNameWithType) {
+ public PauseStatus resumeConsumption(String tableNameWithType, @Nullable String offsetCriteria) {
IdealState updatedIdealState = updatePauseStatusInIdealState(tableNameWithType, false);
// trigger realtime segment validation job to resume consumption
Map<String, String> taskProperties = new HashMap<>();
taskProperties.put(RealtimeSegmentValidationManager.RECREATE_DELETED_CONSUMING_SEGMENT_KEY, "true");
+ if (offsetCriteria != null) {
+ taskProperties.put(RealtimeSegmentValidationManager.OFFSET_CRITERIA, offsetCriteria);
+ }
_helixResourceManager
.invokeControllerPeriodicTask(tableNameWithType, Constants.REALTIME_SEGMENT_VALIDATION_MANAGER, taskProperties);
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java
index b772fcabd1..90c23361d1 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java
@@ -34,6 +34,7 @@ import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
import org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask;
import org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager;
import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.stream.OffsetCriteria;
import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
@@ -55,6 +56,7 @@ public class RealtimeSegmentValidationManager extends ControllerPeriodicTask<Rea
private long _lastSegmentLevelValidationRunTimeMs = 0L;
public static final String RECREATE_DELETED_CONSUMING_SEGMENT_KEY = "recreateDeletedConsumingSegment";
+ public static final String OFFSET_CRITERIA = "offsetCriteria";
public RealtimeSegmentValidationManager(ControllerConf config, PinotHelixResourceManager pinotHelixResourceManager,
LeadControllerManager leadControllerManager, PinotLLCRealtimeSegmentManager llcRealtimeSegmentManager,
@@ -82,7 +84,10 @@ public class RealtimeSegmentValidationManager extends ControllerPeriodicTask<Rea
}
context._recreateDeletedConsumingSegment =
Boolean.parseBoolean(periodicTaskProperties.getProperty(RECREATE_DELETED_CONSUMING_SEGMENT_KEY));
-
+ String offsetCriteriaStr = periodicTaskProperties.getProperty(OFFSET_CRITERIA);
+ if (offsetCriteriaStr != null) {
+ context._offsetCriteria = new OffsetCriteria.OffsetCriteriaBuilder().withOffsetString(offsetCriteriaStr);
+ }
return context;
}
@@ -106,7 +111,7 @@ public class RealtimeSegmentValidationManager extends ControllerPeriodicTask<Rea
if (streamConfig.hasLowLevelConsumerType()) {
_llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
- context._recreateDeletedConsumingSegment);
+ context._recreateDeletedConsumingSegment, context._offsetCriteria);
}
}
@@ -178,6 +183,7 @@ public class RealtimeSegmentValidationManager extends ControllerPeriodicTask<Rea
public static final class Context {
private boolean _runSegmentLevelValidation;
private boolean _recreateDeletedConsumingSegment;
+ private OffsetCriteria _offsetCriteria;
}
@VisibleForTesting
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java
index 14ee8970ec..eee1f2a8a8 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java
@@ -888,7 +888,8 @@ public class PinotLLCRealtimeSegmentManagerTest {
// Expected
}
try {
- segmentManager.ensureAllPartitionsConsuming(segmentManager._tableConfig, segmentManager._streamConfig, false);
+ segmentManager.ensureAllPartitionsConsuming(segmentManager._tableConfig, segmentManager._streamConfig, false,
+ null);
fail();
} catch (IllegalStateException e) {
// Expected
@@ -1115,7 +1116,7 @@ public class PinotLLCRealtimeSegmentManagerTest {
public void ensureAllPartitionsConsuming() {
ensureAllPartitionsConsuming(_tableConfig, _streamConfig, _idealState,
- getNewPartitionGroupMetadataList(_streamConfig, Collections.emptyList()), false);
+ getNewPartitionGroupMetadataList(_streamConfig, Collections.emptyList()), false, null);
}
@Override
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[pinot] 04/04: Include fmpp plugin module inside the src as it's required for building source code (#9321)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.11-rc
in repository https://gitbox.apache.org/repos/asf/pinot.git
commit 24b92eaf0ba9d693320bda22ba5a1e0f60c2e3c1
Author: Xiang Fu <xi...@gmail.com>
AuthorDate: Thu Sep 1 12:46:28 2022 -0700
Include fmpp plugin module inside the src as it's required for building source code (#9321)
---
pinot-distribution/pinot-source-assembly.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pinot-distribution/pinot-source-assembly.xml b/pinot-distribution/pinot-source-assembly.xml
index 6fdec82c7f..6e9c45099e 100644
--- a/pinot-distribution/pinot-source-assembly.xml
+++ b/pinot-distribution/pinot-source-assembly.xml
@@ -53,7 +53,7 @@
<!-- Do not include docker, kubernetes related files -->
<exclude>kubernetes/**</exclude>
<exclude>docker/**</exclude>
- <exclude>contrib/**</exclude>
+ <exclude>contrib/pinot-druid-benchmark/**</exclude>
<exclude>thirdeye/**</exclude>
</excludes>
</fileSet>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org