You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "AnthonyCrxHexa (via GitHub)" <gi...@apache.org> on 2023/02/14 10:50:26 UTC
[GitHub] [druid] AnthonyCrxHexa commented on issue #8858: export-metadata throws java.lang.NullPointerException
AnthonyCrxHexa commented on issue #8858:
URL: https://github.com/apache/druid/issues/8858#issuecomment-1429518254
> When the exception is thrown, only the following files are being created in the specified output directory:
>
> ```
> bash-4.2# ls
> druid_dataSource.csv druid_dataSource_raw.csv druid_segments.csv druid_segments_raw.csv
> ```
>
> Probably interesting, while `druid_segments.csv` contains the correctly rewritten loadspec, I can see `druid_segments_raw.csv` has the old locations specified if I decode the hexblob using Pythons `binascii.unhexlify()`.
> I'm trying to migrate my segments from local storage to S3. When running `export-metadata` with parameters `-b`/`-k` (rewriting deep storage location to S3), it crashes while exporting the segments table. Exporting without rewriting works fine.
>
> I suspect it might be due to the fact that not all segments in the table refer to the same storage location. Recently created and new segments are actually stored in the target bucket already (storage location in `common/runtime.conf` was changed to reflect the S3 configuration).
>
> ### Affected Version
> 0.16
>
> ### Description
> ```
> bash-4.2# java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[] org.apache.druid.cli.Main tools export-metadata --connectURI "jdbc:derby://localhost:1527/var/druid/metadata.db;" -o /tmp -b druid-bucket -k druid/segments
> Nov 12, 2019 10:11:27 AM org.hibernate.validator.internal.util.Version <clinit>
> INFO: HV000001: Hibernate Validator 5.1.3.Final
> 2019-11-12T10:11:27,504 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.common.config.NullValueHandlingConfig] from props[druid.generic.] as [org.apache.druid.common.config.NullValueHandlingConfig@3da05287]
> 2019-11-12T10:11:27,552 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', useExtensionClassloaderFirst=false, hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[]}]
> 2019-11-12T10:11:27,640 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}]
> 2019-11-12T10:11:27,795 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.aws.AWSModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
> 2019-11-12T10:11:27,797 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.gcp.GcpModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
> 2019-11-12T10:11:28,042 INFO [main] org.apache.druid.guice.Jerseys - Adding Jersey resource: org.apache.druid.server.StatusResource
> 2019-11-12T10:11:28,476 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[interface org.apache.druid.server.security.Escalator] from props[druid.escalator.] as [NoopEscalator{}]
> 2019-11-12T10:11:28,486 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.curator.CuratorConfig] from props[druid.zk.service.] as [org.apache.druid.curator.CuratorConfig@3015db78]
> 2019-11-12T10:11:28,490 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [org.apache.druid.curator.ExhibitorConfig@6b760460]
> 2019-11-12T10:11:28,543 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode
> 2019-11-12T10:11:28,545 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
> 2019-11-12T10:11:28,583 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [org.apache.druid.server.initialization.ZkPathsConfig@58d3f4be]
> 2019-11-12T10:11:28,591 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain=null, authorizers=null, unsecuredPaths=[], allowUnauthenticatedHttpOptions=false}]
> 2019-11-12T10:11:28,642 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [org.apache.druid.server.metrics.DruidMonitorSchedulerConfig@7bc6d27a]
> 2019-11-12T10:11:28,646 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]
> 2019-11-12T10:11:28,647 INFO [main] org.apache.druid.server.emitter.EmitterModule - Underlying emitter for ServiceEmitter: org.apache.druid.java.util.emitter.core.NoopEmitter@1ac45389
> 2019-11-12T10:11:28,647 INFO [main] org.apache.druid.server.emitter.EmitterModule - Extra service dimensions: {version=0.16.0-incubating}
> 2019-11-12T10:11:28,648 INFO [main] org.apache.druid.server.metrics.MetricsModule - Adding monitor[org.apache.druid.query.ExecutorServiceMonitor@10bd9df0]
> 2019-11-12T10:11:28,648 INFO [main] org.apache.druid.server.metrics.MetricsModule - Adding monitor[org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@732c9b5c]
> 2019-11-12T10:11:28,720 INFO [main] org.apache.druid.metadata.storage.derby.DerbyConnector - Derby connector instantiated with metadata storage [org.apache.druid.metadata.NoopMetadataStorageProvider$1].
> 2019-11-12T10:11:28,720 INFO [main] org.apache.druid.cli.ExportMetadata - Exporting datasource table: druid_dataSource
> 2019-11-12T10:11:28,862 INFO [main] org.apache.druid.cli.ExportMetadata - Exporting segments table: druid_segments
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.druid.cli.ExportMetadata.getUniqueIDFromLocalLoadSpec(ExportMetadata.java:537)
> at org.apache.druid.cli.ExportMetadata.makePayloadWithConvertedLoadSpec(ExportMetadata.java:429)
> at org.apache.druid.cli.ExportMetadata.rewriteSegmentsExport(ExportMetadata.java:407)
> at org.apache.druid.cli.ExportMetadata.run(ExportMetadata.java:225)
> at org.apache.druid.cli.Main.main(Main.java:113)
> ```
HI, exact same problem here on latest version, did you figured it out ? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org