You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Pavel Stepanov (Jira)" <ji...@apache.org> on 2025/01/21 14:11:00 UTC
[jira] [Updated] (HDDS-12116) Customizable Protobuf shaded prefix in ozonefs-hadoop3-client
[ https://issues.apache.org/jira/browse/HDDS-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Stepanov updated HDDS-12116:
----------------------------------
Description:
I tried running Trino together with Ozone over OFS but encountered a Protobuf shading incompatibility between the [Trino Hadoop library|https://github.com/trinodb/trino-hadoop-apache/tree/master] and the Ozone Hadoop client:
{code:java}
2024-12-27T09:09:30.178Z DEBUG hive-hive-0 org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem Ozone URI for OFS initialization is ofs://ozone-om-1
2024-12-27T09:09:30.183Z WARN hive-hive-0 org.apache.hadoop.ozone.om.helpers.OzoneFSUtils Ignoring ozone.fs.hsync.enabled = true because HBase enhancements are disallowed. To enable it, set ozone.client.hbase.enhancements.allowed = true as well.
2024-12-27T09:09:30.184Z DEBUG hive-hive-0 org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem hsyncEnabled = false
2024-12-27T09:09:30.473Z INFO dispatcher-query-3 io.trino.event.QueryMonitor TIMELINE: Query 20241227_090929_00003_x4ukx :: FINISHED :: elapsed 480ms :: planning 134ms :: waiting 23ms :: scheduling 202ms :: running 128ms :: finishing 16ms :: begin 2024-12-27T09:09:29.959Z :: end
2024-12-27T09:09:30.641Z INFO hive-hive-0 org.apache.hadoop.hdds.utils.LeakDetector Starting leak detector thread OzoneClientObject0.
2024-12-27T09:09:30.816Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.protocolPB.OmTransportFactory Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
2024-12-27T09:09:31.241Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM om1: class org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest cannot be cast to class io.trino.hadoop.$internal.com.google.protobuf.Message (org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest and io.trino.hadoop.$internal.com.google.protobuf.Message are in unnamed module of loader io.trino.filesystem.manager.HdfsClassLoader @43fffec2)
2024-12-27T09:09:31.243Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Incrementing OM proxy index to 0, nodeId: om1
2024-12-27T09:09:33.246Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Failing over OM from om1:0 to om1:0
2024-12-27T09:09:33.246Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM om1: null{code}
The simplest way to resolve this issue is to adjust the ozonefs-hadoop3-client module, which already shades Protobuf, but currently uses a fixed prefix. I suggest making the prefix configurable as a separate property to allow customization if needed.
With the proposed changes, running
{code:java}
mvn clean package -Dproto.shaded.prefix='io.trino.hadoop.\$internal' {code}
inside the ozonefs-hadoop3-client module will properly shade Protobuf and allow the output JAR to be used with Trino.
Related Trino issue: [https://github.com/trinodb/trino/discussions/18026]
was:
I tried running Trino together with Ozone over OFS but encountered a Protobuf shading incompatibility between the [Trino Hadoop library|https://github.com/trinodb/trino-hadoop-apache/tree/master] and the Ozone Hadoop client:
{code:java}
2024-12-27T09:09:30.178Z DEBUG hive-hive-0 org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem Ozone URI for OFS initialization is ofs://ozone-om-1
2024-12-27T09:09:30.183Z WARN hive-hive-0 org.apache.hadoop.ozone.om.helpers.OzoneFSUtils Ignoring ozone.fs.hsync.enabled = true because HBase enhancements are disallowed. To enable it, set ozone.client.hbase.enhancements.allowed = true as well.
2024-12-27T09:09:30.184Z DEBUG hive-hive-0 org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem hsyncEnabled = false
2024-12-27T09:09:30.473Z INFO dispatcher-query-3 io.trino.event.QueryMonitor TIMELINE: Query 20241227_090929_00003_x4ukx :: FINISHED :: elapsed 480ms :: planning 134ms :: waiting 23ms :: scheduling 202ms :: running 128ms :: finishing 16ms :: begin 2024-12-27T09:09:29.959Z :: end
2024-12-27T09:09:30.641Z INFO hive-hive-0 org.apache.hadoop.hdds.utils.LeakDetector Starting leak detector thread OzoneClientObject0.
2024-12-27T09:09:30.816Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.protocolPB.OmTransportFactory Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
2024-12-27T09:09:31.241Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM om1: class org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest cannot be cast to class io.trino.hadoop.$internal.com.google.protobuf.Message (org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest and io.trino.hadoop.$internal.com.google.protobuf.Message are in unnamed module of loader io.trino.filesystem.manager.HdfsClassLoader @43fffec2)
2024-12-27T09:09:31.243Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Incrementing OM proxy index to 0, nodeId: om1
2024-12-27T09:09:33.246Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Failing over OM from om1:0 to om1:0
2024-12-27T09:09:33.246Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM om1: null{code}
The simplest way to resolve this issue is to adjust the ozonefs-hadoop3-client module, which already shades Protobuf, but currently uses a fixed prefix. I suggest making the prefix configurable as a separate property to allow customization if needed.
With the proposed changes, running
{code:java}
mvn clean package -Dproto.shaded.prefix='io.trino.hadoop.\$internal' {code}
inside the ozonefs-hadoop3-client module will properly shade Protobuf and allow the output JAR to be used with Trino.
Related trino issue: https://github.com/trinodb/trino/discussions/18026
> Customizable Protobuf shaded prefix in ozonefs-hadoop3-client
> -------------------------------------------------------------
>
> Key: HDDS-12116
> URL: https://issues.apache.org/jira/browse/HDDS-12116
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Affects Versions: 2.0.0
> Reporter: Pavel Stepanov
> Assignee: Pavel Stepanov
> Priority: Major
>
> I tried running Trino together with Ozone over OFS but encountered a Protobuf shading incompatibility between the [Trino Hadoop library|https://github.com/trinodb/trino-hadoop-apache/tree/master] and the Ozone Hadoop client:
> {code:java}
> 2024-12-27T09:09:30.178Z DEBUG hive-hive-0 org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem Ozone URI for OFS initialization is ofs://ozone-om-1
> 2024-12-27T09:09:30.183Z WARN hive-hive-0 org.apache.hadoop.ozone.om.helpers.OzoneFSUtils Ignoring ozone.fs.hsync.enabled = true because HBase enhancements are disallowed. To enable it, set ozone.client.hbase.enhancements.allowed = true as well.
> 2024-12-27T09:09:30.184Z DEBUG hive-hive-0 org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem hsyncEnabled = false
> 2024-12-27T09:09:30.473Z INFO dispatcher-query-3 io.trino.event.QueryMonitor TIMELINE: Query 20241227_090929_00003_x4ukx :: FINISHED :: elapsed 480ms :: planning 134ms :: waiting 23ms :: scheduling 202ms :: running 128ms :: finishing 16ms :: begin 2024-12-27T09:09:29.959Z :: end
> 2024-12-27T09:09:30.641Z INFO hive-hive-0 org.apache.hadoop.hdds.utils.LeakDetector Starting leak detector thread OzoneClientObject0.
> 2024-12-27T09:09:30.816Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.protocolPB.OmTransportFactory Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
> 2024-12-27T09:09:31.241Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM om1: class org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest cannot be cast to class io.trino.hadoop.$internal.com.google.protobuf.Message (org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest and io.trino.hadoop.$internal.com.google.protobuf.Message are in unnamed module of loader io.trino.filesystem.manager.HdfsClassLoader @43fffec2)
> 2024-12-27T09:09:31.243Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Incrementing OM proxy index to 0, nodeId: om1
> 2024-12-27T09:09:33.246Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Failing over OM from om1:0 to om1:0
> 2024-12-27T09:09:33.246Z DEBUG hive-hive-0 org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM om1: null{code}
> The simplest way to resolve this issue is to adjust the ozonefs-hadoop3-client module, which already shades Protobuf, but currently uses a fixed prefix. I suggest making the prefix configurable as a separate property to allow customization if needed.
> With the proposed changes, running
> {code:java}
> mvn clean package -Dproto.shaded.prefix='io.trino.hadoop.\$internal' {code}
> inside the ozonefs-hadoop3-client module will properly shade Protobuf and allow the output JAR to be used with Trino.
> Related Trino issue: [https://github.com/trinodb/trino/discussions/18026]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org