You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "SendDreams (via GitHub)" <gi...@apache.org> on 2023/07/13 03:19:36 UTC
[GitHub] [hudi] SendDreams opened a new issue, #9180: Hudi metadata to hive metastore Get the parquet schema for this table looking at the latest commit OOM
SendDreams opened a new issue, #9180:
URL: https://github.com/apache/hudi/issues/9180
Trying to sync hoodie table acs_camain_huo with base path hdfs://GWNS:8020/user/hudi/acs_camain_hu of type MERGE_ON_READ
[INFO ] 2023-07-13 11:09:14,065 method:org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:599)
Connected to metastore.
[INFO ] 2023-07-13 11:09:14,065 method:org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:156)
Syncing target hoodie table with hive table(ods.acs_wffins_hu). Hive metastore URL from HiveConf:thrift://bdp-utility-p-0001:9083). Hive metastore URL from HiveSyncConfig:thrift://bdp-utility-p-0001:9083, basePath :hdfs://GWNS:8020/user/hudi/acs_wffins_hu
[INFO ] 2023-07-13 11:09:14,066 method:org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:214)
Trying to sync hoodie table acs_wffins_huo with base path hdfs://GWNS:8020/user/hudi/acs_wffins_hu of type MERGE_ON_READ
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/benpin_report_hu_real_job_02_oom ...
Unable to create /tmp/benpin_report_hu_real_job_02_oom: File exists
[INFO ] 2023-07-13 11:09:15,248 method:org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:628)
Closed a connection to metastore, current connections: 2
[ERROR] 2023-07-13 11:09:15,249 method:org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:140)
Executor executes action [sync hive metadata for instant 20230713110913478] error
java.lang.OutOfMemoryError: Java heap space
at org.apache.hudi.common.table.log.block.HoodieLogBlock.tryReadContent(HoodieLogBlock.java:267)
at org.apache.hudi.common.table.log.HoodieLogFileReader.readBlock(HoodieLogFileReader.java:194)
at org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:411)
at org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:70)
at org.apache.hudi.common.table.TableSchemaResolver.readSchemaFromLogFile(TableSchemaResolver.java:375)
at org.apache.hudi.common.table.TableSchemaResolver.readSchemaFromLogFile(TableSchemaResolver.java:363)
at org.apache.hudi.common.table.TableSchemaResolver.fetchSchemaFromFiles(TableSchemaResolver.java:511)
at org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchemaFromDataFile(TableSchemaResolver.java:272)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:116)
at org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:444)
at org.apache.hudi.common.table.TableSchemaResolver$$Lambda$1800/1082959049.get(Unknown Source)
at org.apache.hudi.util.Lazy.get(Lazy.java:54)
at org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromLatestCommitMetadata(TableSchemaResolver.java:229)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaInternal(TableSchemaResolver.java:197)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:137)
at org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchema(TableSchemaResolver.java:177)
at org.apache.hudi.sync.common.HoodieSyncClient.getStorageSchema(HoodieSyncClient.java:110)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:238)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:188)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:162)
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:340)
at org.apache.hudi.sink.StreamWriteOperatorCoordinator$$Lambda$1789/2130655350.run(Unknown Source)
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
at org.apache.hudi.sink.utils.NonThrownExecutor$$Lambda$1443/2107592834.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
??????:{ "at": { "atMobiles":[ "null" ], "isAtAll": true }, "text": { "content":"flink??[benpin_report_hu_real_job_02]???????" }, "msgtype":"text"}
{"errcode":300001,"errmsg":"????: robot ????????:??? token ?????"}
[INFO ] 2023-07-13 11:09:15,805 method:org.apache.hudi.hive.ddl.HMSDDLExecutor.getTableSchema(HMSDDLExecutor.java:191)
Time taken to getTableSchema: 10 ms
[INFO ] 2023-07-13 11:09:15,802 method:org.apache.hudi.hive.ddl.HMSDDLExecutor.getTableSchema(HMSDDLExecutor.java:191)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SendDreams commented on issue #9180: Hudi metadata to hive metastore Get the parquet schema for this table looking at the latest commit OOM
Posted by "SendDreams (via GitHub)" <gi...@apache.org>.
SendDreams commented on issue #9180:
URL: https://github.com/apache/hudi/issues/9180#issuecomment-1633492408
Is there any god can help me anwser zhe question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi metadata to hive metastore Get the parquet schema for this table looking at the latest commit OOM [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #9180:
URL: https://github.com/apache/hudi/issues/9180#issuecomment-3224173702
Hi @SendDreams Please let me know if you need any help with this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9180: Hudi metadata to hive metastore Get the parquet schema for this table looking at the latest commit OOM
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9180:
URL: https://github.com/apache/hudi/issues/9180#issuecomment-1633515696
@SendDreams Are you running hive sync job? This is failing with Java Heap Space issue. You might consider trying increasing the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi metadata to hive metastore Get the parquet schema for this table looking at the latest commit OOM [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9180:
URL: https://github.com/apache/hudi/issues/9180#issuecomment-1748412533
@SendDreams Were you able to get it resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org