You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Justin Bertram (Jira)" <ji...@apache.org> on 2022/09/28 18:58:00 UTC
[jira] [Comment Edited] (ARTEMIS-4019) Journal export runs out of memory
[ https://issues.apache.org/jira/browse/ARTEMIS-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610693#comment-17610693 ]
Justin Bertram edited comment on ARTEMIS-4019 at 9/28/22 6:57 PM:
------------------------------------------------------------------
It looks like you're executing the {{data exp}} command from the broker's *home* directory. You should use the broker's *instance* directory instead which will give you this:
{noformat}
$ ./artemis help data
NAME
artemis data - data tools group
(print|imp|exp|encode|decode|compact|recover) (example ./artemis data
print)
SYNOPSIS
artemis data
artemis data compact [--verbose] [--broker <brokerConfig>]
[--paging <paging>] [--large-messages <largeMessges>]
[--bindings <binding>] [--journal <journal>]
artemis data decode [--directory <directory>] [--verbose]
[--prefix <prefix>] [--broker <brokerConfig>] [--file-size <size>]
[--paging <paging>] [--large-messages <largeMessges>]
[--suffix <suffix>] [--bindings <binding>] --input <input>
[--journal <journal>]
artemis data encode [--verbose] [--prefix <prefix>]
[--broker <brokerConfig>] [--file-size <size>] [--paging <paging>]
[--large-messages <largeMessges>] [--suffix <suffix>]
[--bindings <binding>] [--journal <journal>] [--directory <directory>]
artemis data exp [--verbose] [--broker <brokerConfig>]
[--paging <paging>] [--large-messages <largeMessges>]
[--jdbc-bindings-table-name <jdbcBindings>] [--f]
[--jdbc-large-message-table-name <jdbcLargeMessages>]
[--jdbc-connection-url <jdbcURL>]
[--jdbc-node-manager-table-name <jdbcNodeManager>]
[--jdbc-driver-class-name <jdbcClassName>]
[--jdbc-page-store-table-name <jdbcPageStore>] [--jdbc]
[--jdbc-message-table-name <jdbcMessages>] [--bindings <binding>]
[--output <output>] [--journal <journal>]
artemis data imp [--password <password>] [--verbose] [--transaction]
[--legacy-prefixes] [--port <port>] [--user <user>] --input <input>
[--host <host>] [--sort]
artemis data print [--max-pages <maxPages>] [--safe] [--skip-bindings]
[--verbose] [--broker <brokerConfig>] [--reclaimed] [--paging <paging>]
[--large-messages <largeMessges>]
[--jdbc-bindings-table-name <jdbcBindings>] [--f]
[--jdbc-large-message-table-name <jdbcLargeMessages>]
[--jdbc-connection-url <jdbcURL>]
[--jdbc-node-manager-table-name <jdbcNodeManager>]
[--jdbc-driver-class-name <jdbcClassName>]
[--jdbc-page-store-table-name <jdbcPageStore>] [--jdbc]
[--jdbc-message-table-name <jdbcMessages>] [--skip-journal]
[--bindings <binding>] [--output <output>] [--journal <journal>]
artemis data recover [--verbose] [--broker <brokerConfig>]
[--paging <paging>] [--large-messages <largeMessges>]
[--jdbc-bindings-table-name <jdbcBindings>] [--f]
[--jdbc-large-message-table-name <jdbcLargeMessages>]
--target <outputJournal> [--jdbc-connection-url <jdbcURL>]
[--jdbc-node-manager-table-name <jdbcNodeManager>]
[--jdbc-driver-class-name <jdbcClassName>] [--reclaimed]
[--jdbc-page-store-table-name <jdbcPageStore>] [--jdbc]
[--jdbc-message-table-name <jdbcMessages>] [--bindings <binding>]
[--output <output>] [--journal <journal>]
COMMANDS
With no arguments, Display help information
recover
Recover (undelete) every message on the journal by creating a new
output journal. Rolled backed and acked messages will be sent out to
the output as much as possible.
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --jdbc-bindings-table-name option, Name of the jdbc bindigns
table
With --f option, This will allow certain tools like print-data to be
performed ignoring any running servers. WARNING: Changing data
concurrently with a running broker may damage your data. Be careful
with this option.
With --jdbc-large-message-table-name option, Name of the large
messages table
With --target option, Output folder container the new journal with
all the generated messages
With --jdbc-connection-url option, The connection used for the
database
With --jdbc-node-manager-table-name option, Name of the jdbc node
manager table
With --jdbc-driver-class-name option, JDBC driver classname
With --reclaimed option, This option will try to recover as many
records as possible from reclaimed files
With --jdbc-page-store-table-name option, Name of the page store
messages table
With --jdbc option, It will activate jdbc
With --jdbc-message-table-name option, Name of the jdbc messages
table
With --bindings option, The folder used for bindings (default from
broker.xml)
With --output option, Output name for the file
With --journal option, The folder used for messages journal (default
from broker.xml)
print
Print data records information (WARNING: don't use while a
production server is running)
With --max-pages option, It will print your data structure without
showing your data
With --safe option, It will print your data structure without
showing your data
With --skip-bindings option, It will ignore printing the bindings
journal
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --reclaimed option, This option will try to print as many
records as possible from reclaimed files
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --jdbc-bindings-table-name option, Name of the jdbc bindigns
table
With --f option, This will allow certain tools like print-data to be
performed ignoring any running servers. WARNING: Changing data
concurrently with a running broker may damage your data. Be careful
with this option.
With --jdbc-large-message-table-name option, Name of the large
messages table
With --jdbc-connection-url option, The connection used for the
database
With --jdbc-node-manager-table-name option, Name of the jdbc node
manager table
With --jdbc-driver-class-name option, JDBC driver classname
With --jdbc-page-store-table-name option, Name of the page store
messages table
With --jdbc option, It will activate jdbc
With --jdbc-message-table-name option, Name of the jdbc messages
table
With --skip-journal option, It will ignore printing the messages
journal
With --bindings option, The folder used for bindings (default from
broker.xml)
With --output option, Output name for the file
With --journal option, The folder used for messages journal (default
from broker.xml)
exp
Export all message-data using an XML that could be interpreted by
any system.
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --jdbc-bindings-table-name option, Name of the jdbc bindigns
table
With --f option, This will allow certain tools like print-data to be
performed ignoring any running servers. WARNING: Changing data
concurrently with a running broker may damage your data. Be careful
with this option.
With --jdbc-large-message-table-name option, Name of the large
messages table
With --jdbc-connection-url option, The connection used for the
database
With --jdbc-node-manager-table-name option, Name of the jdbc node
manager table
With --jdbc-driver-class-name option, JDBC driver classname
With --jdbc-page-store-table-name option, Name of the page store
messages table
With --jdbc option, It will activate jdbc
With --jdbc-message-table-name option, Name of the jdbc messages
table
With --bindings option, The folder used for bindings (default from
broker.xml)
With --output option, Output name for the file
With --journal option, The folder used for messages journal (default
from broker.xml)
imp
Import all message-data using an XML that could be interpreted by
any system.
With --password option, User name used to import the data. (default
null)
With --verbose option, Adds more information on the execution
With --transaction option, If this is set to true you will need a
whole transaction to commit at the end. (default false)
With --legacy-prefixes option, Do not remove prefixes from legacy
imports
With --port option, The port used to import the data (default 61616)
With --user option, User name used to import the data. (default
null)
With --input option, The input file name (default=exp.dmp)
With --host option, The host used to import the data (default
localhost)
With --sort option, Sort the messages from the input (used for older
versions that won't sort messages)
decode
Decode a journal's internal format into a new journal set of files
With --directory option, The journal folder (default journal folder
from broker.xml)
With --verbose option, Adds more information on the execution
With --prefix option, The journal prefix (default activemq-data)
With --broker option, This would override the broker configuration
from the bootstrap
With --file-size option, The journal size (default 10485760)
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --suffix option, The journal suffix (default amq)
With --bindings option, The folder used for bindings (default from
broker.xml)
With --input option, The input file name (default=exp.dmp)
With --journal option, The folder used for messages journal (default
from broker.xml)
encode
Encode a set of journal files into an internal encoded data format
With --verbose option, Adds more information on the execution
With --prefix option, The journal prefix (default activemq-data)
With --broker option, This would override the broker configuration
from the bootstrap
With --file-size option, The journal size (default 10485760)
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --suffix option, The journal suffix (default amq)
With --bindings option, The folder used for bindings (default from
broker.xml)
With --journal option, The folder used for messages journal (default
from broker.xml)
With --directory option, The journal folder (default the journal
folder from broker.xml)
compact
Compacts the journal of a non running server
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --bindings option, The folder used for bindings (default from
broker.xml)
With --journal option, The folder used for messages journal (default
from broker.xml){noformat}
was (Author: jbertram):
It looks like you're executing the {{data exp}} command from the broker's **home** directory. You should use the broker's **instance** directory instead which will give you this:
{noformat}
$ ./artemis help data
NAME
artemis data - data tools group
(print|imp|exp|encode|decode|compact|recover) (example ./artemis data
print)
SYNOPSIS
artemis data
artemis data compact [--verbose] [--broker <brokerConfig>]
[--paging <paging>] [--large-messages <largeMessges>]
[--bindings <binding>] [--journal <journal>]
artemis data decode [--directory <directory>] [--verbose]
[--prefix <prefix>] [--broker <brokerConfig>] [--file-size <size>]
[--paging <paging>] [--large-messages <largeMessges>]
[--suffix <suffix>] [--bindings <binding>] --input <input>
[--journal <journal>]
artemis data encode [--verbose] [--prefix <prefix>]
[--broker <brokerConfig>] [--file-size <size>] [--paging <paging>]
[--large-messages <largeMessges>] [--suffix <suffix>]
[--bindings <binding>] [--journal <journal>] [--directory <directory>]
artemis data exp [--verbose] [--broker <brokerConfig>]
[--paging <paging>] [--large-messages <largeMessges>]
[--jdbc-bindings-table-name <jdbcBindings>] [--f]
[--jdbc-large-message-table-name <jdbcLargeMessages>]
[--jdbc-connection-url <jdbcURL>]
[--jdbc-node-manager-table-name <jdbcNodeManager>]
[--jdbc-driver-class-name <jdbcClassName>]
[--jdbc-page-store-table-name <jdbcPageStore>] [--jdbc]
[--jdbc-message-table-name <jdbcMessages>] [--bindings <binding>]
[--output <output>] [--journal <journal>]
artemis data imp [--password <password>] [--verbose] [--transaction]
[--legacy-prefixes] [--port <port>] [--user <user>] --input <input>
[--host <host>] [--sort]
artemis data print [--max-pages <maxPages>] [--safe] [--skip-bindings]
[--verbose] [--broker <brokerConfig>] [--reclaimed] [--paging <paging>]
[--large-messages <largeMessges>]
[--jdbc-bindings-table-name <jdbcBindings>] [--f]
[--jdbc-large-message-table-name <jdbcLargeMessages>]
[--jdbc-connection-url <jdbcURL>]
[--jdbc-node-manager-table-name <jdbcNodeManager>]
[--jdbc-driver-class-name <jdbcClassName>]
[--jdbc-page-store-table-name <jdbcPageStore>] [--jdbc]
[--jdbc-message-table-name <jdbcMessages>] [--skip-journal]
[--bindings <binding>] [--output <output>] [--journal <journal>]
artemis data recover [--verbose] [--broker <brokerConfig>]
[--paging <paging>] [--large-messages <largeMessges>]
[--jdbc-bindings-table-name <jdbcBindings>] [--f]
[--jdbc-large-message-table-name <jdbcLargeMessages>]
--target <outputJournal> [--jdbc-connection-url <jdbcURL>]
[--jdbc-node-manager-table-name <jdbcNodeManager>]
[--jdbc-driver-class-name <jdbcClassName>] [--reclaimed]
[--jdbc-page-store-table-name <jdbcPageStore>] [--jdbc]
[--jdbc-message-table-name <jdbcMessages>] [--bindings <binding>]
[--output <output>] [--journal <journal>]
COMMANDS
With no arguments, Display help information
recover
Recover (undelete) every message on the journal by creating a new
output journal. Rolled backed and acked messages will be sent out to
the output as much as possible.
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --jdbc-bindings-table-name option, Name of the jdbc bindigns
table
With --f option, This will allow certain tools like print-data to be
performed ignoring any running servers. WARNING: Changing data
concurrently with a running broker may damage your data. Be careful
with this option.
With --jdbc-large-message-table-name option, Name of the large
messages table
With --target option, Output folder container the new journal with
all the generated messages
With --jdbc-connection-url option, The connection used for the
database
With --jdbc-node-manager-table-name option, Name of the jdbc node
manager table
With --jdbc-driver-class-name option, JDBC driver classname
With --reclaimed option, This option will try to recover as many
records as possible from reclaimed files
With --jdbc-page-store-table-name option, Name of the page store
messages table
With --jdbc option, It will activate jdbc
With --jdbc-message-table-name option, Name of the jdbc messages
table
With --bindings option, The folder used for bindings (default from
broker.xml)
With --output option, Output name for the file
With --journal option, The folder used for messages journal (default
from broker.xml)
print
Print data records information (WARNING: don't use while a
production server is running)
With --max-pages option, It will print your data structure without
showing your data
With --safe option, It will print your data structure without
showing your data
With --skip-bindings option, It will ignore printing the bindings
journal
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --reclaimed option, This option will try to print as many
records as possible from reclaimed files
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --jdbc-bindings-table-name option, Name of the jdbc bindigns
table
With --f option, This will allow certain tools like print-data to be
performed ignoring any running servers. WARNING: Changing data
concurrently with a running broker may damage your data. Be careful
with this option.
With --jdbc-large-message-table-name option, Name of the large
messages table
With --jdbc-connection-url option, The connection used for the
database
With --jdbc-node-manager-table-name option, Name of the jdbc node
manager table
With --jdbc-driver-class-name option, JDBC driver classname
With --jdbc-page-store-table-name option, Name of the page store
messages table
With --jdbc option, It will activate jdbc
With --jdbc-message-table-name option, Name of the jdbc messages
table
With --skip-journal option, It will ignore printing the messages
journal
With --bindings option, The folder used for bindings (default from
broker.xml)
With --output option, Output name for the file
With --journal option, The folder used for messages journal (default
from broker.xml)
exp
Export all message-data using an XML that could be interpreted by
any system.
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --jdbc-bindings-table-name option, Name of the jdbc bindigns
table
With --f option, This will allow certain tools like print-data to be
performed ignoring any running servers. WARNING: Changing data
concurrently with a running broker may damage your data. Be careful
with this option.
With --jdbc-large-message-table-name option, Name of the large
messages table
With --jdbc-connection-url option, The connection used for the
database
With --jdbc-node-manager-table-name option, Name of the jdbc node
manager table
With --jdbc-driver-class-name option, JDBC driver classname
With --jdbc-page-store-table-name option, Name of the page store
messages table
With --jdbc option, It will activate jdbc
With --jdbc-message-table-name option, Name of the jdbc messages
table
With --bindings option, The folder used for bindings (default from
broker.xml)
With --output option, Output name for the file
With --journal option, The folder used for messages journal (default
from broker.xml)
imp
Import all message-data using an XML that could be interpreted by
any system.
With --password option, User name used to import the data. (default
null)
With --verbose option, Adds more information on the execution
With --transaction option, If this is set to true you will need a
whole transaction to commit at the end. (default false)
With --legacy-prefixes option, Do not remove prefixes from legacy
imports
With --port option, The port used to import the data (default 61616)
With --user option, User name used to import the data. (default
null)
With --input option, The input file name (default=exp.dmp)
With --host option, The host used to import the data (default
localhost)
With --sort option, Sort the messages from the input (used for older
versions that won't sort messages)
decode
Decode a journal's internal format into a new journal set of files
With --directory option, The journal folder (default journal folder
from broker.xml)
With --verbose option, Adds more information on the execution
With --prefix option, The journal prefix (default activemq-data)
With --broker option, This would override the broker configuration
from the bootstrap
With --file-size option, The journal size (default 10485760)
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --suffix option, The journal suffix (default amq)
With --bindings option, The folder used for bindings (default from
broker.xml)
With --input option, The input file name (default=exp.dmp)
With --journal option, The folder used for messages journal (default
from broker.xml)
encode
Encode a set of journal files into an internal encoded data format
With --verbose option, Adds more information on the execution
With --prefix option, The journal prefix (default activemq-data)
With --broker option, This would override the broker configuration
from the bootstrap
With --file-size option, The journal size (default 10485760)
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --suffix option, The journal suffix (default amq)
With --bindings option, The folder used for bindings (default from
broker.xml)
With --journal option, The folder used for messages journal (default
from broker.xml)
With --directory option, The journal folder (default the journal
folder from broker.xml)
compact
Compacts the journal of a non running server
With --verbose option, Adds more information on the execution
With --broker option, This would override the broker configuration
from the bootstrap
With --paging option, The folder used for paging (default from
broker.xml)
With --large-messages option, The folder used for large-messages
(default from broker.xml)
With --bindings option, The folder used for bindings (default from
broker.xml)
With --journal option, The folder used for messages journal (default
from broker.xml){noformat}
> Journal export runs out of memory
> ----------------------------------
>
> Key: ARTEMIS-4019
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4019
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Environment: SUSE Linux Enterprise Server 15 SP3
> JAVA_ARGS="-XX:+PrintClassHistogram -XX:+UseG1GC -XX:+UseStringDeduplication -Xms512M -Xmx4G -Dhawtio.disableProxy=true -Dhawtio.realm=activemq -Dhawtio.offline=true -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml "
> Reporter: Peter Machon
> Priority: Major
>
> To avoid the situation from ARTEMIS-4018 on other nodes in our environment we wanted to export the old journal and reimport it before restarting the upgraded version. The export however runs OOM:
> {noformat}
> /var/opt/apache-artemis/integrationbus>bin/artemis data exp --output /var/opt/apache-artemis/integrationbus/etc/data3.exp
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at io.netty.buffer.UnpooledByteBufAllocator.newHeapBuffer(UnpooledByteBufAllocator.java:82)
> at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:169)
> at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160)
> at io.netty.buffer.Unpooled.buffer(Unpooled.java:119)
> at org.apache.activemq.artemis.core.message.impl.CoreMessage.initBuffer(CoreMessage.java:156)
> at org.apache.activemq.artemis.core.message.impl.CoreMessage.reloadPersistence(CoreMessage.java:1252)
> at org.apache.activemq.artemis.core.message.impl.CoreMessagePersister.decode(CoreMessagePersister.java:85)
> at org.apache.activemq.artemis.core.message.impl.CoreMessagePersister.decode(CoreMessagePersister.java:28)
> at org.apache.activemq.artemis.spi.core.protocol.MessagePersister.decode(MessagePersister.java:120)
> at org.apache.activemq.artemis.core.persistence.impl.journal.DescribeJournal.newObjectEncoding(DescribeJournal.java:623)
> at org.apache.activemq.artemis.cli.commands.tools.xml.XmlDataExporter.processMessageJournal(XmlDataExporter.java:218)
> at org.apache.activemq.artemis.cli.commands.tools.xml.XmlDataExporter.writeXMLData(XmlDataExporter.java:150)
> at org.apache.activemq.artemis.cli.commands.tools.xml.XmlDataExporter.writeOutput(XmlDataExporter.java:144)
> at org.apache.activemq.artemis.cli.commands.tools.xml.XmlDataExporter.process(XmlDataExporter.java:133)
> at org.apache.activemq.artemis.cli.commands.tools.xml.XmlDataExporter.execute(XmlDataExporter.java:100)
> at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:160)
> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:108)
> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:135)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:134)
> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:50){noformat}
> Note: We were attempting to export the old journal with the new broker version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)