You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@inlong.apache.org by Goson zhang <go...@apache.org> on 2022/03/25 02:26:00 UTC
Re: [DISCUSS] About the removal of berkeleydb-je component
Hi all:
Through the construction of [1], we have completed the optional storage
implementation of the Master's metadata between BDB and ZooKeeper. Users
only need to update the master.ini configuration according to the guidance
document of [2] and select the corresponding [meta_zookeeper] or [meta_bdb]
configuration unit is enough; before and after the migration, users can
export and import the configured metadata through the
bin/tubemq-metadata-bru.sh script to complete the metadata migration
operation.
The BDB component is an embedded storage component. When using it, the
system must use the basic classes in this component to complete related
data read and write operations. In TubeMQ, the content of the BDB component
is only used, with no code copying, and no API modification. Therefore, by
modifying the TubeMQ system this time, all BDB-related code content can be
eliminated and exist independently.
For compatibility reasons, we still retain the BDB part of the system,
because we used BDB in the previous version, there must be users who do not
want to make major adjustments and want to continue to use BDB as a storage
medium. We leave this option to users.
If you find any new problems during use, please let us know through issue
or email, and let's see what the problem is together.
Thanks!
1. https://github.com/apache/incubator-inlong/issues/2282
2.
https://inlong.apache.org/docs/next/modules/tubemq/configure_introduction
Goson zhang <go...@apache.org> 于2022年1月23日周日 16:11写道：
> I have submitted an issue [1] for processing and tracking this thread, and
> keeping track of related content through this issue.
>
> Thanks!
>
> 1. https://github.com/apache/incubator-inlong/issues/2282
>
> Goson zhang <go...@apache.org> 于2022年1月20日周四 10:16写道：
>
>> Someone gave me a point to consider the compatibility of the system
>> switching from berkeleydb-je to ZooKeeper.
>>
>> I am going to provide tools and help documents so that users can use
>> tools to read metadata from the system running on berkeleydb-je, and then
>> import it into the new version system based on ZooKeeper.
>>
>> And I am wondering if the storage implementation of berkeleydb-je can be
>> kept in the project so that everyone can choose by themselves.
>> I understand that after the metadata storage based on ZooKeeper is
>> implemented, the storage of berkeleydb-je has become a non-mandatory
>> dependency.
>>
>> After the implementation is completed, I will ask the mentors for
>> suggestions based on the implementation.
>>
>> Thanks!
>>
>> Goson zhang <go...@apache.org> 于2022年1月17日周一 11:32写道：
>>
>>> Hi all:
>>>
>>> According to Apache's protocol requirements[1], berkeleydb-je is not a
>>> dependent component that meets Apache's open-source requirements. Next, I
>>> plan to replace it, before finding a better metadata storage component,
>>> with ZooKeeper.
>>>
>>> Let's know your consideration, in order to avoid the impact before and
>>> after the change, we have abstracted the operation of metadata in the early
>>> stage, so that the module can seamlessly migrate this part of the code.
>>>
>>> Thanks!
>>>
>>> 1. https://lists.apache.org/thread/nchbf7zyg78lwh04op2k0634csj8p5yv
>>>
>>
Re: [DISCUSS] About the removal of berkeleydb-je component
Posted by Goson zhang <go...@apache.org>.
I modified the master.ini configuration and pom.xml dependencies of the
TubeMQ module, set ZooKeeper as the default storage component, set the
dependency scope of com.sleepycat.je-7.3.7.jar to "provided", and updated
the configuration and usage Notes on the guidance documentation; the TubeMQ
component now uses ZooKeeper by default to store metadata.
For users who upgraded versions before 1.1.0, and users of 1.1.0 and later
versions, they need to download the com.sleepycat.je-7.3.7.jar package by
themselves, otherwise, the system will report
"java.lang.ClassNotFoundException:
com.sleepycat.je.ReplicaConsistencyPolicy" error, which you only need to
download the dependency package and put it in the lib directory of the
module
You are welcome to give feedback on new problems found during use at any
time.
Thanks!
Goson zhang <go...@apache.org> 于2022年3月25日周五 10:26写道：
> Hi all:
>
> Through the construction of [1], we have completed the optional storage
> implementation of the Master's metadata between BDB and ZooKeeper. Users
> only need to update the master.ini configuration according to the guidance
> document of [2] and select the corresponding [meta_zookeeper] or [meta_bdb]
> configuration unit is enough; before and after the migration, users can
> export and import the configured metadata through the
> bin/tubemq-metadata-bru.sh script to complete the metadata migration
> operation.
>
> The BDB component is an embedded storage component. When using it, the
> system must use the basic classes in this component to complete related
> data read and write operations. In TubeMQ, the content of the BDB component
> is only used, with no code copying, and no API modification. Therefore, by
> modifying the TubeMQ system this time, all BDB-related code content can be
> eliminated and exist independently.
>
> For compatibility reasons, we still retain the BDB part of the system,
> because we used BDB in the previous version, there must be users who do not
> want to make major adjustments and want to continue to use BDB as a storage
> medium. We leave this option to users.
>
> If you find any new problems during use, please let us know through issue
> or email, and let's see what the problem is together.
>
> Thanks!
>
>
> 1. https://github.com/apache/incubator-inlong/issues/2282
> 2.
> https://inlong.apache.org/docs/next/modules/tubemq/configure_introduction
>
> Goson zhang <go...@apache.org> 于2022年1月23日周日 16:11写道：
>
>> I have submitted an issue [1] for processing and tracking this thread,
>> and keeping track of related content through this issue.
>>
>> Thanks!
>>
>> 1. https://github.com/apache/incubator-inlong/issues/2282
>>
>> Goson zhang <go...@apache.org> 于2022年1月20日周四 10:16写道：
>>
>>> Someone gave me a point to consider the compatibility of the system
>>> switching from berkeleydb-je to ZooKeeper.
>>>
>>> I am going to provide tools and help documents so that users can use
>>> tools to read metadata from the system running on berkeleydb-je, and then
>>> import it into the new version system based on ZooKeeper.
>>>
>>> And I am wondering if the storage implementation of berkeleydb-je can be
>>> kept in the project so that everyone can choose by themselves.
>>> I understand that after the metadata storage based on ZooKeeper is
>>> implemented, the storage of berkeleydb-je has become a non-mandatory
>>> dependency.
>>>
>>> After the implementation is completed, I will ask the mentors for
>>> suggestions based on the implementation.
>>>
>>> Thanks!
>>>
>>> Goson zhang <go...@apache.org> 于2022年1月17日周一 11:32写道：
>>>
>>>> Hi all:
>>>>
>>>> According to Apache's protocol requirements[1], berkeleydb-je is not a
>>>> dependent component that meets Apache's open-source requirements. Next, I
>>>> plan to replace it, before finding a better metadata storage component,
>>>> with ZooKeeper.
>>>>
>>>> Let's know your consideration, in order to avoid the impact before and
>>>> after the change, we have abstracted the operation of metadata in the early
>>>> stage, so that the module can seamlessly migrate this part of the code.
>>>>
>>>> Thanks!
>>>>
>>>> 1. https://lists.apache.org/thread/nchbf7zyg78lwh04op2k0634csj8p5yv
>>>>
>>>