You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/13 07:51:16 UTC
[GitHub] [iceberg] openinx opened a new pull request #3725: Aliyun: Add iceberg-aliyun to engines runtime
openinx opened a new pull request #3725:
URL: https://github.com/apache/iceberg/pull/3725
## Flink1.13 Example:
* Start the Flink SQL cilent:
```bash
wget https://gosspublic.alicdn.com/sdks/java/aliyun_java_sdk_3.10.2.zip
unzip aliyun_java_sdk_3.10.2.zip
# Download from https://www.apache.org/dyn/closer.lua/flink/flink-1.13.3/flink-1.13.3-bin-scala_2.12.tgz
FLINK_HOME=/path/to/flink-release
# Downlaod from https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime
FLINK_RUNTIME_LIB=/path/to/iceberg-flink-runtime-1.14-0.13.0-SNAPSHOT.jar
# Download from https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6_2.11/1.13.3/flink-sql-connector-hive-2.3.6_2.11-1.13.3.jar
HIVE_METASTORE_LIB=/path/to/flink-sql-connector-hive-2.3.6_2.12-1.13.2.jar
DEPS="-j $FLINK_RUNTIME_LIB -j $HIVE_METASTORE_LIB"
for dep in `find aliyun_java_sdk_3.10.2 -type f -name '*.jar'`; do
DEPS="$DEPS -j $dep"
done
eval "$FLINK_HOME/bin/sql-client.sh embedded $DEPS shell"
```
* Execute the flink SQL:
```sql
CREATE CATALOG hive WITH (
'type' = 'iceberg',
'uri' = 'thrift://localhost:9083',
'warehouse' = 'oss://iceberg-test/warehouse',
'io-impl' = 'org.apache.iceberg.aliyun.oss.OSSFileIO',
'oss.endpoint' = 'oss-cn-hangzhou.aliyuncs.com',
'client.access-key-id' = '******',
'client.access-key-secret' = '******'
);
CREATE TABLE `hive`.`default`.`sample` (
id BIGINT,
data STRING
) WITH (
'engine.hive.enabled' = 'true'
);
INSERT INTO `hive`.`default`.`sample` VALUES (1, 'AAA');
SELECT * FROM sample;
+----+------+
| id | data |
+----+------+
| 1 | AAA |
+----+------+
1 row in set
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on pull request #3725: Aliyun: Add iceberg-aliyun to engines runtime
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on pull request #3725:
URL: https://github.com/apache/iceberg/pull/3725#issuecomment-994100261
As discussed in dev list, I think we are preferring this option, marking it as a part of 0.13 release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3725: Aliyun: Add iceberg-aliyun to engines runtime
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3725:
URL: https://github.com/apache/iceberg/pull/3725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #3725: Aliyun: Add iceberg-aliyun to engines runtime
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #3725:
URL: https://github.com/apache/iceberg/pull/3725#issuecomment-992843587
Am I correct in thinking that aliyun comes with a bit more dependencies than some of the other projects, like `'org.glassfish.jaxb:jaxb-runtime'`?
Because they're `compileOnly` we can safely say that, it won't cause users issues if they also have the same thing on their class path but with a highly different version (if they're not using Aliyun, that is)?
EDIT: I believe I was thinking of spring framework, which is actually test only. Nevermind!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #3725: Aliyun: Add iceberg-aliyun to engines runtime
Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #3725:
URL: https://github.com/apache/iceberg/pull/3725#issuecomment-993103290
> Am I correct in thinking that aliyun comes with a bit more dependencies than some of the other projects, like 'org.glassfish.jaxb:jaxb-runtime'?
The `compileOnly` org.glassfish.jaxb:jaxb-runtime was added for building iceberg-aliyun module for jdk9 or higher, because aliyun-oss-sdk depends on those libraries to parse/write XML body. Those libraries was always included in jd8 or before, but was removed since jdk9. For more details please read this: https://www.alibabacloud.com/help/doc-detail/32009.html
Thanks for your comment, @kbendick .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3725: Aliyun: Add iceberg-aliyun to engines runtime
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3725:
URL: https://github.com/apache/iceberg/pull/3725#issuecomment-994153227
I checked the dependencies that are included by this and it looks good:
```
./gradlew iceberg-spark:iceberg-spark-runtime-3.2_2.12:dependencies --configuration runtimeClasspath
```
```
...
+--- project :iceberg-aliyun
| +--- project :iceberg-api (*)
| +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
| +--- project :iceberg-bundled-guava
| +--- project :iceberg-core (*)
| \--- project :iceberg-common (*)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3725: Aliyun: Add iceberg-aliyun to engines runtime
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3725:
URL: https://github.com/apache/iceberg/pull/3725#issuecomment-992843587
Am I correct in thinking that aliyun comes with a bit more dependencies than some of the other projects, like `'org.glassfish.jaxb:jaxb-runtime'`?
Because they're `compileOnly` we can safely say that, it won't cause users issues if they also have the same thing on their class path but with a highly different version (if they're not using Aliyun, that is)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org