You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/19 11:32:04 UTC
[GitHub] [hudi] stym06 opened a new issue #4638: [SUPPORT] Deltastreamer job failing with RocksDB exception
stym06 opened a new issue #4638:
URL: https://github.com/apache/hudi/issues/4638
**Describe the problem you faced**
Driver is exiting with below stacktrace while doing a BULK_INSERT from Kafka to Blob
**Environment Description**
* Hudi version : 0.9.0
* Spark version : 2.4.4
* Hive version : 3.1.2
* Hadoop version : 2.7.3
* Storage (HDFS/S3/GCS..) : Azure
* Running on Docker? (yes/no) : K8s
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
22/01/19 16:56:31 INFO RocksDBDAO: From Rocks DB : [db/db_impl_write.cc:1423] [hudi_view_wasb:__<container>@<account>.blob.core.windows.net_data_pipelines_hudi_kafka_telemetrics_v2_dp.hmi.quectel.event.lpe.packet.v2] New memtable created with log file: #19. Immutable memtables: 0.
JniUtil::getJinEnv - Fatal: could not attach current thread to JVM!
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f858a697446, pid=13, tid=0x00007f85b1164b10
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.12.0
# Distribution: Custom build (Mon Aug 12 12:55:38 UTC 2019)
# Problematic frame:
# C [librocksdbjni3568498499091871873.so+0x2c3446] rocksdb::LoggerJniCallback::Logv(rocksdb::InfoLogLevel, char const*, __va_list_tag*)+0x76
#
# Core dump written. Default location: /opt/spark/work-dir/core or core.13
#
# An error report file with more information is saved as:
# /opt/spark/work-dir/hs_err_pid13.log
22/01/19 16:56:31 INFO RocksDbBasedFileSystemView: Finished adding new partition (v3/dt=2022-01-04) to ROCKSDB based file-system view at /tmp/hoodie_timeline_rocksdb, Total file-groups=3817
22/01/19 16:56:31 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=3818, NumFileGroups=3817, FileGroupsCreationTime=218, StoreTimeTaken=787
22/01/19 16:56:31 INFO RocksDBDAO: Prefix Search for (query=type=slice,part=v3/dt=2022-01-04,id=) on hudi_view_wasb:__oemdpv3prd-v1@oemdpv3prd.blob.core.windows.net_data_pipelines_hudi_kafka_telemetrics_v2_dp.hmi.quectel.event.lpe.packet.v2. Total Time Taken (msec)=16. Serialization Time taken(micro)=9889, num entries=3817
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# https://icedtea.classpath.org/bugzilla
#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4638: [SUPPORT] Deltastreamer job failing with RocksDB exception
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4638:
URL: https://github.com/apache/hudi/issues/4638#issuecomment-1018524555
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4638: [SUPPORT] Deltastreamer job failing with RocksDB exception
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4638:
URL: https://github.com/apache/hudi/issues/4638#issuecomment-1017015954
cool, thanks for the update. Would you mind putting up a patch with the version upgrade. I can help review and land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 edited a comment on issue #4638: [SUPPORT] Deltastreamer job failing with RocksDB exception
Posted by GitBox <gi...@apache.org>.
stym06 edited a comment on issue #4638:
URL: https://github.com/apache/hudi/issues/4638#issuecomment-1016574839
On quick googling, found that this is being caused by a segmentation fault in the LoggerJniCallback class inside rocksdb when the logging class of rocksdb which is being called here:https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java#L102
Hudi is using rocksdb-5.17.2 which has this issue and fails when run on an Alpine based image/vm which is stated here: https://github.com/facebook/rocksdb/issues/4277#issuecomment-533453851
The fix is available with rocksdb v6.27.3 onwards. (https://github.com/facebook/rocksdb/commit/2367656b6c19048d76037d24025ef0caab136866#diff-950c1d37d965830d86d07977aab5a4d84550e86930fc2c6dd0ec8fc40a799b49)
After removing the call or upgrading the version, this issue goes away on my local build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4638: [SUPPORT] Deltastreamer job failing with RocksDB exception
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4638:
URL: https://github.com/apache/hudi/issues/4638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 commented on issue #4638: [SUPPORT] Deltastreamer job failing with RocksDB exception
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #4638:
URL: https://github.com/apache/hudi/issues/4638#issuecomment-1018314013
Filed a PR: https://github.com/apache/hudi/pull/4663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 commented on issue #4638: [SUPPORT] Deltastreamer job failing with RocksDB exception
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #4638:
URL: https://github.com/apache/hudi/issues/4638#issuecomment-1016574839
On quick googling, found that this is being caused by a segmentation fault in the LoggerJniCallback class inside rocksdb when the logging class of rocksdb which is being called here:https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java#L102
Hudi is using rocksdb-5.17.2 which has this issue and fails when run on an Alpine based image/vm which is stated here: https://github.com/facebook/rocksdb/issues/4277#issuecomment-533453851
The fix is available with rocksdb v6.27.3 onwards. (https://github.com/facebook/rocksdb/commit/2367656b6c19048d76037d24025ef0caab136866#diff-950c1d37d965830d86d07977aab5a4d84550e86930fc2c6dd0ec8fc40a799b49)
After upgrading the version, this issue goes away on my local build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org