You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bryanck (via GitHub)" <gi...@apache.org> on 2024/07/21 21:46:05 UTC
[PR] Kafka Connect: Runtime distribution and integration tests [iceberg]
bryanck opened a new pull request, #10739:
URL: https://github.com/apache/iceberg/pull/10739
This PR is the next stage in submitting the Kafka Connect Iceberg sink connector, and is a follow up to https://github.com/apache/iceberg/pull/8701, https://github.com/apache/iceberg/pull/9466, https://github.com/apache/iceberg/pull/9641, and https://github.com/apache/iceberg/pull/10351.
This adds building of runtime distributions plus some integration tests. There are two runtime distributions, one that bundles the HMS client and related libraries, and one that does not. The one without the HMS client is much smaller and also doesn't have dependencies with critical security vulnerabilities. Repositories like Confluent Hub require connectors be free of critical vulnerabilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1688393002
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
Review Comment:
That's perfect. My docker engine didn't start so it appeared that I needed to start it manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution and integration tests [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#issuecomment-2242818777
> This adds building of runtime distributions plus some integration tests
Can we please split into two PRs. one for adding runtime distributions and one for integration tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687396082
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
Review Comment:
This does run automatically, using TestContainer's `ComposeContainer`. This allows us to also use this to start up a local environment for manual testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687428234
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
Review Comment:
(FYI, it starts this [here](https://github.com/bryanck/iceberg/blob/8d39497a801b610b3050b8afa632a674cc2c2f0d/kafka-connect/kafka-connect-runtime/src/integration/java/org/apache/iceberg/connect/TestContext.java#L52))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1690897385
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
Should we really add this dependency?
I am not sure about the future roadmap of this project.
Can we use some OSS implementation of the catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691439004
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
I'm not sure if there is a better alternative but I'm happy to switch if there is one. @danielcweeks do you have concerns relying on this image?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1686952595
##########
kafka-connect/kafka-connect-runtime/src/main/resources/manifest.json:
##########
@@ -0,0 +1,47 @@
+{
+ "title": "Apache Iceberg Sink Connector",
+ "name": "iceberg-kafka-connect",
+ "logo": "assets/iceberg.png",
Review Comment:
Does this image exist? I see a png in the docs, but I don't think the paths correspond.
##########
kafka-connect/build.gradle:
##########
@@ -57,3 +57,187 @@ project(":iceberg-kafka-connect:iceberg-kafka-connect") {
useJUnitPlatform()
}
}
+
+project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
+ apply plugin: 'distribution'
+
+ configurations {
+ hive {
+ extendsFrom runtimeClasspath
+ }
+ all {
+ exclude group: 'javax.activation', module: 'activation'
+ // force upgrades for dependencies with known vulnerabilities...
+ resolutionStrategy {
+ force 'org.codehaus.jettison:jettison:1.5.4'
+ force 'org.xerial.snappy:snappy-java:1.1.10.5'
+ force 'org.apache.commons:commons-compress:1.26.0'
+ force 'org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.2.0'
+ }
+ }
+ }
+
+ sourceSets {
+ integration {
+ java.srcDir "$projectDir/src/integration/java"
+ resources.srcDir "$projectDir/src/integration/resources"
+ }
+ }
+
+ configurations {
+ integrationImplementation.extendsFrom testImplementation
+ integrationRuntime.extendsFrom testRuntimeOnly
+ }
+
+ dependencies {
+ implementation project(':iceberg-kafka-connect:iceberg-kafka-connect')
+ implementation(libs.hadoop3.common) {
+ exclude group: 'log4j'
+ exclude group: 'org.slf4j'
+ exclude group: 'ch.qos.reload4j'
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'com.fasterxml.woodstox'
+ exclude group: 'com.google.guava'
+ exclude group: 'com.google.protobuf'
+ exclude group: 'org.apache.curator'
+ exclude group: 'org.apache.zookeeper'
+ exclude group: 'org.apache.kerby'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-auth'
+ exclude group: 'org.apache.commons', module: 'commons-configuration2'
+ exclude group: 'org.apache.hadoop.thirdparty', module: 'hadoop-shaded-protobuf_3_7'
+ exclude group: 'org.codehaus.woodstox'
+ exclude group: 'org.eclipse.jetty'
+ }
+ implementation project(':iceberg-orc')
+ implementation project(':iceberg-parquet')
+
+ implementation project(':iceberg-aws')
+ implementation platform(libs.awssdk.bom)
+ implementation 'software.amazon.awssdk:apache-client'
+ implementation 'software.amazon.awssdk:auth'
+ implementation 'software.amazon.awssdk:iam'
+ implementation 'software.amazon.awssdk:sso'
+ implementation 'software.amazon.awssdk:s3'
+ implementation 'software.amazon.awssdk:kms'
+ implementation 'software.amazon.awssdk:glue'
+ implementation 'software.amazon.awssdk:sts'
+ implementation 'software.amazon.awssdk:dynamodb'
+ implementation 'software.amazon.awssdk:lakeformation'
+
+ implementation project(':iceberg-gcp')
+ implementation platform(libs.google.libraries.bom)
+ implementation 'com.google.cloud:google-cloud-storage'
+
+ implementation project(':iceberg-azure')
+ implementation platform(libs.azuresdk.bom)
+ implementation 'com.azure:azure-storage-file-datalake'
+ implementation 'com.azure:azure-identity'
+
+ hive project(':iceberg-hive-metastore')
+ hive(libs.hive2.metastore) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ exclude group: 'org.pentaho' // missing dependency
+ exclude group: 'org.apache.hbase'
+ exclude group: 'org.apache.logging.log4j'
+ exclude group: 'co.cask.tephra'
+ exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ exclude group: 'org.eclipse.jetty.aggregate', module: 'jetty-all'
+ exclude group: 'org.eclipse.jetty.orbit', module: 'javax.servlet'
+ exclude group: 'org.apache.parquet', module: 'parquet-hadoop-bundle'
+ exclude group: 'com.tdunning', module: 'json'
+ exclude group: 'javax.transaction', module: 'transaction-api'
+ exclude group: 'com.zaxxer', module: 'HikariCP'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-common'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-applicationhistoryservice'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-resourcemanager'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-web-proxy'
+ exclude group: 'org.apache.hive', module: 'hive-service-rpc'
+ exclude group: 'com.github.joshelser', module: 'dropwizard-metrics-hadoop-metrics2-reporter'
+ }
+ hive(libs.hadoop3.client) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ }
+
+ integrationImplementation project(':iceberg-api')
+ integrationImplementation project(':iceberg-common')
+ integrationImplementation project(':iceberg-core')
+ integrationImplementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
+ integrationImplementation platform(libs.jackson.bom)
+ integrationImplementation libs.jackson.core
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.kafka.clients
+ integrationImplementation libs.kafka.connect.api
+ integrationImplementation libs.kafka.connect.json
+ integrationImplementation libs.testcontainers
+ integrationImplementation libs.httpcomponents.httpclient5
+ integrationImplementation libs.awaitility
+ }
+
+ task integrationTest(type: Test) {
+ useJUnitPlatform()
+ testClassesDirs = sourceSets.integration.output.classesDirs
+ classpath = sourceSets.integration.runtimeClasspath
+ jvmArgs += project.property('extraJvmArgs')
+ }
+
+ processResources {
+ filter {
+ it.replace('__VERSION__', project.version.toString())
+ }
+ }
+
+ distributions {
+ main {
+ contents {
+ from(processResources.destinationDir) {
+ include 'manifest.json'
+ }
+ into('lib/') {
+ from configurations.runtimeClasspath
+ }
+ into('doc/') {
+ from '$rootDir/LICENSE'
+ from '$rootDir/README.md'
+ }
+ into('assets/') {
+ from '$rootDir/logos'
+ }
+ }
+ }
+ hive {
+ contents {
+ from(processResources.destinationDir) {
+ include 'manifest.json'
+ }
+ into('lib/') {
+ from configurations.hive
+ }
+ into('doc/') {
+ from '$rootDir/LICENSE'
+ from '$rootDir/README.md'
+ }
+ into('assets/') {
+ from '$rootDir/logos'
Review Comment:
I don't think this is producing anything in the distribution.
##########
kafka-connect/build.gradle:
##########
@@ -57,3 +57,187 @@ project(":iceberg-kafka-connect:iceberg-kafka-connect") {
useJUnitPlatform()
}
}
+
+project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
+ apply plugin: 'distribution'
+
+ configurations {
+ hive {
+ extendsFrom runtimeClasspath
+ }
+ all {
+ exclude group: 'javax.activation', module: 'activation'
+ // force upgrades for dependencies with known vulnerabilities...
+ resolutionStrategy {
+ force 'org.codehaus.jettison:jettison:1.5.4'
+ force 'org.xerial.snappy:snappy-java:1.1.10.5'
+ force 'org.apache.commons:commons-compress:1.26.0'
+ force 'org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.2.0'
+ }
+ }
+ }
+
+ sourceSets {
+ integration {
+ java.srcDir "$projectDir/src/integration/java"
+ resources.srcDir "$projectDir/src/integration/resources"
+ }
+ }
+
+ configurations {
+ integrationImplementation.extendsFrom testImplementation
+ integrationRuntime.extendsFrom testRuntimeOnly
+ }
+
+ dependencies {
+ implementation project(':iceberg-kafka-connect:iceberg-kafka-connect')
+ implementation(libs.hadoop3.common) {
+ exclude group: 'log4j'
+ exclude group: 'org.slf4j'
+ exclude group: 'ch.qos.reload4j'
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'com.fasterxml.woodstox'
+ exclude group: 'com.google.guava'
+ exclude group: 'com.google.protobuf'
+ exclude group: 'org.apache.curator'
+ exclude group: 'org.apache.zookeeper'
+ exclude group: 'org.apache.kerby'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-auth'
+ exclude group: 'org.apache.commons', module: 'commons-configuration2'
+ exclude group: 'org.apache.hadoop.thirdparty', module: 'hadoop-shaded-protobuf_3_7'
+ exclude group: 'org.codehaus.woodstox'
+ exclude group: 'org.eclipse.jetty'
+ }
+ implementation project(':iceberg-orc')
+ implementation project(':iceberg-parquet')
+
+ implementation project(':iceberg-aws')
+ implementation platform(libs.awssdk.bom)
+ implementation 'software.amazon.awssdk:apache-client'
+ implementation 'software.amazon.awssdk:auth'
+ implementation 'software.amazon.awssdk:iam'
+ implementation 'software.amazon.awssdk:sso'
+ implementation 'software.amazon.awssdk:s3'
+ implementation 'software.amazon.awssdk:kms'
+ implementation 'software.amazon.awssdk:glue'
+ implementation 'software.amazon.awssdk:sts'
+ implementation 'software.amazon.awssdk:dynamodb'
+ implementation 'software.amazon.awssdk:lakeformation'
+
+ implementation project(':iceberg-gcp')
+ implementation platform(libs.google.libraries.bom)
+ implementation 'com.google.cloud:google-cloud-storage'
+
+ implementation project(':iceberg-azure')
+ implementation platform(libs.azuresdk.bom)
+ implementation 'com.azure:azure-storage-file-datalake'
+ implementation 'com.azure:azure-identity'
+
+ hive project(':iceberg-hive-metastore')
+ hive(libs.hive2.metastore) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ exclude group: 'org.pentaho' // missing dependency
+ exclude group: 'org.apache.hbase'
+ exclude group: 'org.apache.logging.log4j'
+ exclude group: 'co.cask.tephra'
+ exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ exclude group: 'org.eclipse.jetty.aggregate', module: 'jetty-all'
+ exclude group: 'org.eclipse.jetty.orbit', module: 'javax.servlet'
+ exclude group: 'org.apache.parquet', module: 'parquet-hadoop-bundle'
+ exclude group: 'com.tdunning', module: 'json'
+ exclude group: 'javax.transaction', module: 'transaction-api'
+ exclude group: 'com.zaxxer', module: 'HikariCP'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-common'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-applicationhistoryservice'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-resourcemanager'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-web-proxy'
+ exclude group: 'org.apache.hive', module: 'hive-service-rpc'
+ exclude group: 'com.github.joshelser', module: 'dropwizard-metrics-hadoop-metrics2-reporter'
+ }
+ hive(libs.hadoop3.client) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ }
+
+ integrationImplementation project(':iceberg-api')
+ integrationImplementation project(':iceberg-common')
+ integrationImplementation project(':iceberg-core')
+ integrationImplementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
+ integrationImplementation platform(libs.jackson.bom)
+ integrationImplementation libs.jackson.core
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.kafka.clients
+ integrationImplementation libs.kafka.connect.api
+ integrationImplementation libs.kafka.connect.json
+ integrationImplementation libs.testcontainers
+ integrationImplementation libs.httpcomponents.httpclient5
+ integrationImplementation libs.awaitility
+ }
+
+ task integrationTest(type: Test) {
+ useJUnitPlatform()
+ testClassesDirs = sourceSets.integration.output.classesDirs
+ classpath = sourceSets.integration.runtimeClasspath
+ jvmArgs += project.property('extraJvmArgs')
+ }
+
+ processResources {
+ filter {
+ it.replace('__VERSION__', project.version.toString())
+ }
+ }
+
+ distributions {
+ main {
+ contents {
+ from(processResources.destinationDir) {
+ include 'manifest.json'
+ }
+ into('lib/') {
+ from configurations.runtimeClasspath
+ }
+ into('doc/') {
+ from '$rootDir/LICENSE'
+ from '$rootDir/README.md'
+ }
+ into('assets/') {
+ from '$rootDir/logos'
Review Comment:
I don't think this is producing anything in the distribution.
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ - MINIO_DOMAIN=minio
+ ports:
+ - 9001:9001
+ - 9000:9000
+ command: ["server", "/data", "--console-address", ":9001"]
+
+ iceberg:
+ image: tabulario/iceberg-rest
+ depends_on:
+ - minio
+ hostname: iceberg
+ ports:
+ - 8181:8181
+ environment:
+ - AWS_REGION=us-east-1
+ - CATALOG_WAREHOUSE=s3://bucket/warehouse/
+ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
+ - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_PATH__STYLE__ACCESS=true
+ - CATALOG_S3_ACCESS__KEY__ID=minioadmin
+ - CATALOG_S3_SECRET__ACCESS__KEY=minioadmin
+
+ zookeeper:
Review Comment:
I haven't looked recently, but I think they're recommending KRaft now, which would get ZK out of the config. That might be nice if it's supported via the docker image.
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
Review Comment:
I feel like we either need a way to run these automatically alongside the tests (e.g. TestContainers) or have a README with the commands to setup/teardown the compose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687394776
##########
kafka-connect/kafka-connect-runtime/src/main/resources/manifest.json:
##########
@@ -0,0 +1,47 @@
+{
+ "title": "Apache Iceberg Sink Connector",
+ "name": "iceberg-kafka-connect",
+ "logo": "assets/iceberg.png",
Review Comment:
No, my mistake, this is checked in now, and I fixed the paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691726716
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
Thanks for the update.
It would be nice to maintain and publish this image under OSS iceberg account 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution and integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#issuecomment-2242960006
The integration tests depend on the runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687396397
##########
kafka-connect/build.gradle:
##########
@@ -57,3 +57,187 @@ project(":iceberg-kafka-connect:iceberg-kafka-connect") {
useJUnitPlatform()
}
}
+
+project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
+ apply plugin: 'distribution'
+
+ configurations {
+ hive {
+ extendsFrom runtimeClasspath
+ }
+ all {
+ exclude group: 'javax.activation', module: 'activation'
+ // force upgrades for dependencies with known vulnerabilities...
+ resolutionStrategy {
+ force 'org.codehaus.jettison:jettison:1.5.4'
+ force 'org.xerial.snappy:snappy-java:1.1.10.5'
+ force 'org.apache.commons:commons-compress:1.26.0'
+ force 'org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.2.0'
+ }
+ }
+ }
+
+ sourceSets {
+ integration {
+ java.srcDir "$projectDir/src/integration/java"
+ resources.srcDir "$projectDir/src/integration/resources"
+ }
+ }
+
+ configurations {
+ integrationImplementation.extendsFrom testImplementation
+ integrationRuntime.extendsFrom testRuntimeOnly
+ }
+
+ dependencies {
+ implementation project(':iceberg-kafka-connect:iceberg-kafka-connect')
+ implementation(libs.hadoop3.common) {
+ exclude group: 'log4j'
+ exclude group: 'org.slf4j'
+ exclude group: 'ch.qos.reload4j'
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'com.fasterxml.woodstox'
+ exclude group: 'com.google.guava'
+ exclude group: 'com.google.protobuf'
+ exclude group: 'org.apache.curator'
+ exclude group: 'org.apache.zookeeper'
+ exclude group: 'org.apache.kerby'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-auth'
+ exclude group: 'org.apache.commons', module: 'commons-configuration2'
+ exclude group: 'org.apache.hadoop.thirdparty', module: 'hadoop-shaded-protobuf_3_7'
+ exclude group: 'org.codehaus.woodstox'
+ exclude group: 'org.eclipse.jetty'
+ }
+ implementation project(':iceberg-orc')
+ implementation project(':iceberg-parquet')
+
+ implementation project(':iceberg-aws')
+ implementation platform(libs.awssdk.bom)
+ implementation 'software.amazon.awssdk:apache-client'
+ implementation 'software.amazon.awssdk:auth'
+ implementation 'software.amazon.awssdk:iam'
+ implementation 'software.amazon.awssdk:sso'
+ implementation 'software.amazon.awssdk:s3'
+ implementation 'software.amazon.awssdk:kms'
+ implementation 'software.amazon.awssdk:glue'
+ implementation 'software.amazon.awssdk:sts'
+ implementation 'software.amazon.awssdk:dynamodb'
+ implementation 'software.amazon.awssdk:lakeformation'
+
+ implementation project(':iceberg-gcp')
+ implementation platform(libs.google.libraries.bom)
+ implementation 'com.google.cloud:google-cloud-storage'
+
+ implementation project(':iceberg-azure')
+ implementation platform(libs.azuresdk.bom)
+ implementation 'com.azure:azure-storage-file-datalake'
+ implementation 'com.azure:azure-identity'
+
+ hive project(':iceberg-hive-metastore')
+ hive(libs.hive2.metastore) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ exclude group: 'org.pentaho' // missing dependency
+ exclude group: 'org.apache.hbase'
+ exclude group: 'org.apache.logging.log4j'
+ exclude group: 'co.cask.tephra'
+ exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ exclude group: 'org.eclipse.jetty.aggregate', module: 'jetty-all'
+ exclude group: 'org.eclipse.jetty.orbit', module: 'javax.servlet'
+ exclude group: 'org.apache.parquet', module: 'parquet-hadoop-bundle'
+ exclude group: 'com.tdunning', module: 'json'
+ exclude group: 'javax.transaction', module: 'transaction-api'
+ exclude group: 'com.zaxxer', module: 'HikariCP'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-common'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-applicationhistoryservice'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-resourcemanager'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-web-proxy'
+ exclude group: 'org.apache.hive', module: 'hive-service-rpc'
+ exclude group: 'com.github.joshelser', module: 'dropwizard-metrics-hadoop-metrics2-reporter'
+ }
+ hive(libs.hadoop3.client) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ }
+
+ integrationImplementation project(':iceberg-api')
+ integrationImplementation project(':iceberg-common')
+ integrationImplementation project(':iceberg-core')
+ integrationImplementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
+ integrationImplementation platform(libs.jackson.bom)
+ integrationImplementation libs.jackson.core
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.kafka.clients
+ integrationImplementation libs.kafka.connect.api
+ integrationImplementation libs.kafka.connect.json
+ integrationImplementation libs.testcontainers
+ integrationImplementation libs.httpcomponents.httpclient5
+ integrationImplementation libs.awaitility
+ }
+
+ task integrationTest(type: Test) {
+ useJUnitPlatform()
+ testClassesDirs = sourceSets.integration.output.classesDirs
+ classpath = sourceSets.integration.runtimeClasspath
+ jvmArgs += project.property('extraJvmArgs')
+ }
+
+ processResources {
+ filter {
+ it.replace('__VERSION__', project.version.toString())
+ }
+ }
+
+ distributions {
+ main {
+ contents {
+ from(processResources.destinationDir) {
+ include 'manifest.json'
+ }
+ into('lib/') {
+ from configurations.runtimeClasspath
+ }
+ into('doc/') {
+ from '$rootDir/LICENSE'
+ from '$rootDir/README.md'
+ }
+ into('assets/') {
+ from '$rootDir/logos'
Review Comment:
This is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687396791
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ - MINIO_DOMAIN=minio
+ ports:
+ - 9001:9001
+ - 9000:9000
+ command: ["server", "/data", "--console-address", ":9001"]
+
+ iceberg:
+ image: tabulario/iceberg-rest
+ depends_on:
+ - minio
+ hostname: iceberg
+ ports:
+ - 8181:8181
+ environment:
+ - AWS_REGION=us-east-1
+ - CATALOG_WAREHOUSE=s3://bucket/warehouse/
+ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
+ - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_PATH__STYLE__ACCESS=true
+ - CATALOG_S3_ACCESS__KEY__ID=minioadmin
+ - CATALOG_S3_SECRET__ACCESS__KEY=minioadmin
+
+ zookeeper:
Review Comment:
The config was more complicated, but I'll switch to that, given ZK is deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691637442
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
This is currently used by a number of projects as a test depending (python,rust,trino) and is fine for now. We've discussed moving this into the test fixtures in core and I'll work with @Fokko to get that done and we can move the test dependencies separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687395068
##########
kafka-connect/build.gradle:
##########
@@ -57,3 +57,187 @@ project(":iceberg-kafka-connect:iceberg-kafka-connect") {
useJUnitPlatform()
}
}
+
+project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
+ apply plugin: 'distribution'
+
+ configurations {
+ hive {
+ extendsFrom runtimeClasspath
+ }
+ all {
+ exclude group: 'javax.activation', module: 'activation'
+ // force upgrades for dependencies with known vulnerabilities...
+ resolutionStrategy {
+ force 'org.codehaus.jettison:jettison:1.5.4'
+ force 'org.xerial.snappy:snappy-java:1.1.10.5'
+ force 'org.apache.commons:commons-compress:1.26.0'
+ force 'org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.2.0'
+ }
+ }
+ }
+
+ sourceSets {
+ integration {
+ java.srcDir "$projectDir/src/integration/java"
+ resources.srcDir "$projectDir/src/integration/resources"
+ }
+ }
+
+ configurations {
+ integrationImplementation.extendsFrom testImplementation
+ integrationRuntime.extendsFrom testRuntimeOnly
+ }
+
+ dependencies {
+ implementation project(':iceberg-kafka-connect:iceberg-kafka-connect')
+ implementation(libs.hadoop3.common) {
+ exclude group: 'log4j'
+ exclude group: 'org.slf4j'
+ exclude group: 'ch.qos.reload4j'
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'com.fasterxml.woodstox'
+ exclude group: 'com.google.guava'
+ exclude group: 'com.google.protobuf'
+ exclude group: 'org.apache.curator'
+ exclude group: 'org.apache.zookeeper'
+ exclude group: 'org.apache.kerby'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-auth'
+ exclude group: 'org.apache.commons', module: 'commons-configuration2'
+ exclude group: 'org.apache.hadoop.thirdparty', module: 'hadoop-shaded-protobuf_3_7'
+ exclude group: 'org.codehaus.woodstox'
+ exclude group: 'org.eclipse.jetty'
+ }
+ implementation project(':iceberg-orc')
+ implementation project(':iceberg-parquet')
+
+ implementation project(':iceberg-aws')
+ implementation platform(libs.awssdk.bom)
+ implementation 'software.amazon.awssdk:apache-client'
+ implementation 'software.amazon.awssdk:auth'
+ implementation 'software.amazon.awssdk:iam'
+ implementation 'software.amazon.awssdk:sso'
+ implementation 'software.amazon.awssdk:s3'
+ implementation 'software.amazon.awssdk:kms'
+ implementation 'software.amazon.awssdk:glue'
+ implementation 'software.amazon.awssdk:sts'
+ implementation 'software.amazon.awssdk:dynamodb'
+ implementation 'software.amazon.awssdk:lakeformation'
+
+ implementation project(':iceberg-gcp')
+ implementation platform(libs.google.libraries.bom)
+ implementation 'com.google.cloud:google-cloud-storage'
+
+ implementation project(':iceberg-azure')
+ implementation platform(libs.azuresdk.bom)
+ implementation 'com.azure:azure-storage-file-datalake'
+ implementation 'com.azure:azure-identity'
+
+ hive project(':iceberg-hive-metastore')
+ hive(libs.hive2.metastore) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ exclude group: 'org.pentaho' // missing dependency
+ exclude group: 'org.apache.hbase'
+ exclude group: 'org.apache.logging.log4j'
+ exclude group: 'co.cask.tephra'
+ exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ exclude group: 'org.eclipse.jetty.aggregate', module: 'jetty-all'
+ exclude group: 'org.eclipse.jetty.orbit', module: 'javax.servlet'
+ exclude group: 'org.apache.parquet', module: 'parquet-hadoop-bundle'
+ exclude group: 'com.tdunning', module: 'json'
+ exclude group: 'javax.transaction', module: 'transaction-api'
+ exclude group: 'com.zaxxer', module: 'HikariCP'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-common'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-applicationhistoryservice'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-resourcemanager'
+ exclude group: 'org.apache.hadoop', module: 'hadoop-yarn-server-web-proxy'
+ exclude group: 'org.apache.hive', module: 'hive-service-rpc'
+ exclude group: 'com.github.joshelser', module: 'dropwizard-metrics-hadoop-metrics2-reporter'
+ }
+ hive(libs.hadoop3.client) {
+ exclude group: 'org.apache.avro', module: 'avro'
+ exclude group: 'org.slf4j', module: 'slf4j-log4j12'
+ }
+
+ integrationImplementation project(':iceberg-api')
+ integrationImplementation project(':iceberg-common')
+ integrationImplementation project(':iceberg-core')
+ integrationImplementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
+ integrationImplementation platform(libs.jackson.bom)
+ integrationImplementation libs.jackson.core
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.jackson.databind
+ integrationImplementation libs.kafka.clients
+ integrationImplementation libs.kafka.connect.api
+ integrationImplementation libs.kafka.connect.json
+ integrationImplementation libs.testcontainers
+ integrationImplementation libs.httpcomponents.httpclient5
+ integrationImplementation libs.awaitility
+ }
+
+ task integrationTest(type: Test) {
+ useJUnitPlatform()
+ testClassesDirs = sourceSets.integration.output.classesDirs
+ classpath = sourceSets.integration.runtimeClasspath
+ jvmArgs += project.property('extraJvmArgs')
+ }
+
+ processResources {
+ filter {
+ it.replace('__VERSION__', project.version.toString())
+ }
+ }
+
+ distributions {
+ main {
+ contents {
+ from(processResources.destinationDir) {
+ include 'manifest.json'
+ }
+ into('lib/') {
+ from configurations.runtimeClasspath
+ }
+ into('doc/') {
+ from '$rootDir/LICENSE'
+ from '$rootDir/README.md'
+ }
+ into('assets/') {
+ from '$rootDir/logos'
+ }
+ }
+ }
+ hive {
+ contents {
+ from(processResources.destinationDir) {
+ include 'manifest.json'
+ }
+ into('lib/') {
+ from configurations.hive
+ }
+ into('doc/') {
+ from '$rootDir/LICENSE'
+ from '$rootDir/README.md'
+ }
+ into('assets/') {
+ from '$rootDir/logos'
Review Comment:
Right it wasn't, this should be fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691741295
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
Thanks @jbonofre! That was one of the biggest outstanding issues because we don't currently produce any images as a project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck merged PR #10739:
URL: https://github.com/apache/iceberg/pull/10739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687428234
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
Review Comment:
(FYI, it starts this [here](https://github.com/bryanck/iceberg/blob/01b7d99237500cf19e7b74eaf8cfd051c7638dd5/kafka-connect/kafka-connect-runtime/src/integration/java/org/apache/iceberg/connect/TestContext.java#L59))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691734005
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
I can help on the DockerHub "seat" for apache/iceberg (I just did that for the new openserverless podling). Just let me know 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687396082
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
Review Comment:
This does run automatically, using TestContainer's `ComposeContainer`. This allows us to also use Compose to start up a local environment for manual testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution and integration tests [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#issuecomment-2243015462
> The integration tests depend on the runtime.
I know. We can review and merge the runtime PR first and then rebase the integration test PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution and integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#issuecomment-2243024192
The integration tests are testing the runtime, so I feel they belong together. Also, the runtime build is just the `build.gradle` so this seems unneccessary to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687420089
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ - MINIO_DOMAIN=minio
+ ports:
+ - 9001:9001
+ - 9000:9000
+ command: ["server", "/data", "--console-address", ":9001"]
+
+ iceberg:
+ image: tabulario/iceberg-rest
+ depends_on:
+ - minio
+ hostname: iceberg
+ ports:
+ - 8181:8181
+ environment:
+ - AWS_REGION=us-east-1
+ - CATALOG_WAREHOUSE=s3://bucket/warehouse/
+ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
+ - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_PATH__STYLE__ACCESS=true
+ - CATALOG_S3_ACCESS__KEY__ID=minioadmin
+ - CATALOG_S3_SECRET__ACCESS__KEY=minioadmin
+
+ zookeeper:
Review Comment:
I switched this to KRaft.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1687396791
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ - MINIO_DOMAIN=minio
+ ports:
+ - 9001:9001
+ - 9000:9000
+ command: ["server", "/data", "--console-address", ":9001"]
+
+ iceberg:
+ image: tabulario/iceberg-rest
+ depends_on:
+ - minio
+ hostname: iceberg
+ ports:
+ - 8181:8181
+ environment:
+ - AWS_REGION=us-east-1
+ - CATALOG_WAREHOUSE=s3://bucket/warehouse/
+ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
+ - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_PATH__STYLE__ACCESS=true
+ - CATALOG_S3_ACCESS__KEY__ID=minioadmin
+ - CATALOG_S3_SECRET__ACCESS__KEY=minioadmin
+
+ zookeeper:
Review Comment:
The config was more complicated, but I'll switch to that, given ZK is deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#issuecomment-2256167831
Thanks for the reviews @danielcweeks , @fqaiser94 , and @ajantha-bhat !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691739280
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
Yes.
With that account we can even publish some more test image like Trino, kafaka-connect, Spark etc.
And we can maintain these images in Iceberg repo test module or as a separate subproject "Iceland - A playground for Iceberg"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691755922
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
@danielcweeks happy to help ! 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #10739:
URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691728921
##########
kafka-connect/kafka-connect-runtime/docker/docker-compose.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+volumes:
+ data: {}
+
+services:
+ minio:
+ image: minio/minio
+ hostname: minio
+ environment:
+ - MINIO_ROOT_USER=minioadmin
+ - MINIO_ROOT_PASSWORD=minioadmin
+ ports:
+ - 9000:9000
+ - 9001:9001
+ volumes:
+ - data:/data
+ command: server /data --console-address ":9001"
+
+ create-bucket:
+ image: minio/mc
+ depends_on:
+ - minio
+ volumes:
+ - data:/data
+ entrypoint: mc mb /data/bucket
+
+ iceberg:
+ image: tabulario/iceberg-rest
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org