You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wudongdong00 (via GitHub)" <gi...@apache.org> on 2024/04/10 08:01:44 UTC
[I] [Bug] Remote submission task prompts that s3 credentials cannot be found [paimon]
wudongdong00 opened a new issue, #3191:
URL: https://github.com/apache/paimon/issues/3191
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.7
### Compute Engine
1.14
### Minimal reproduce step
There is no problem submitting in sql-client,But when submitting from java to remote, it will prompt that the s3 certificate cannot be found.
StreamExecutionEnvironment Env = StreamExecutionEnvironment.createRemoteEnvironment("192.168.100.14", 8083); StreamTableEnvironment tEnv = StreamTableEnvironment.create(Env);
tEnv.executeSql(" CREATE CATALOG paimon_catalog WITH (\n" +
" 'type' = 'paimon',\n" +
" 'warehouse' = 's3://paimon-ods/20240118',\n" +
" 's3.endpoint' = 'http://192.168.100.221:9000',\n" +
" 's3.access-key' = 'Minio',\n" +
" 's3.secret-key' = 'Test1234!'\n" +
" )");
tEnv.executeSql("USE CATALOG paimon_catalog");
tEnv.executeSql("CREATE TEMPORARY TABLE t1000 (\n" +
" id BIGINT COMMENT 'id' ,\n" +
" name STRING COMMENT 'name'\n" +
" WITH ( \n" +
"'connector' = 'mysql-cdc',\n" +
"'hostname' = '192.168.100.1',\n" +
"'port' = '3306',\n" +
"'username' = 'root',\n" +
"'password' = 'xxxx',\n" +
"'server-time-zone' = 'Asia/Shanghai',\n" +
"'debezium.snapshot.mode' = 'initial',\n" +
"'database-name' = 'ja',\n" +
"'table-name' = 'a1'\n" +
")");
tEnv.executeSql("CREATE TABLE IF NOT EXISTS my1 (\n" +
" ID BIGINT PRIMARY KEY NOT ENFORCED COMMENT 'ID' ,\n" +
" NAME STRING COMMENT 'NAME' ,\n" +
" ZJHM STRING COMMENT 'ZJHM'\n" +
" )\n" +
" WITH (\n" +
" 'bucket' = '1',\n" +
" 'bucket-key' = 'ID'\n" +
" )");
tEnv.executeSql(" INSERT INTO my1 (\n" +
"ID,\n" +
"NAME)\n" +
" SELECT id,\n" +
"name\n" +
" FROM t1000");
This is the error log ：
2024-04-10 14:52:34,831 DEBUG org.apache.hadoop.security.UserGroupInformation [] - hadoop login
2024-04-10 14:52:34,832 DEBUG org.apache.hadoop.security.UserGroupInformation [] - hadoop login commit
2024-04-10 14:52:34,832 DEBUG org.apache.hadoop.security.UserGroupInformation [] - using local user:UnixPrincipal: root
2024-04-10 14:52:34,832 DEBUG org.apache.hadoop.security.UserGroupInformation [] - Using user: "UnixPrincipal: root" with name root
2024-04-10 14:52:34,832 DEBUG org.apache.hadoop.security.UserGroupInformation [] - User entry: "root"
2024-04-10 14:52:34,832 DEBUG org.apache.hadoop.security.UserGroupInformation [] - UGI loginUser:root (auth:SIMPLE)
2024-04-10 14:52:35,023 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - For URI s3://paimon-ods//20240118/default.db/sink_paimon_table/schema/schema-0, using credentials AWSCredentialProviderList[refcount= 1: [SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, com.amazonaws.auth.InstanceProfileCredentialsProvider@702e2713]
2024-04-10 14:52:35,031 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.connection.maximum is 15
2024-04-10 14:52:35,031 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.attempts.maximum is 20
2024-04-10 14:52:35,031 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.connection.establish.timeout is 5000
2024-04-10 14:52:35,031 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.connection.timeout is 200000
2024-04-10 14:52:35,031 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.socket.send.buffer is 8192
2024-04-10 14:52:35,032 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.socket.recv.buffer is 8192
2024-04-10 14:52:35,035 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Using User-Agent: Hadoop 3.2.2
2024-04-10 14:52:35,327 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - Enabling path style access!
2024-04-10 14:52:35,333 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.paging.maximum is 5000
2024-04-10 14:52:35,334 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.block.size is 33554432
2024-04-10 14:52:35,334 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.readahead.range is 65536
2024-04-10 14:52:35,334 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.max.total.tasks is 5
2024-04-10 14:52:35,335 DEBUG org.apache.hadoop.fs.s3a.S3AUtils [] - Value of fs.s3a.threads.keepalivetime is 60
2024-04-10 14:52:35,360 DEBUG org.apache.hadoop.fs.s3a.AWSCredentialProviderList [] - No credentials provided by SimpleAWSCredentialsProvider: org.apache.hadoop.fs.s3a.CredentialInitializationException: Access key or secret key is unset
### What doesn't meet your expectations?
Can be successfully submitted
### Anything else?
no
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Remote submission task prompts that s3 credentials cannot be found [paimon]
Posted by "wudongdong00 (via GitHub)" <gi...@apache.org>.
wudongdong00 closed issue #3191: [Bug] Remote submission task prompts that s3 credentials cannot be found
URL: https://github.com/apache/paimon/issues/3191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org