You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Duo Zhang (Jira)" <ji...@apache.org> on 2021/12/02 16:15:00 UTC
[jira] [Commented] (HBASE-26259) Fallback support to pure Java compression
[ https://issues.apache.org/jira/browse/HBASE-26259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452484#comment-17452484 ]
Duo Zhang commented on HBASE-26259:
-----------------------------------
Seems the commit here breaks the branch-2 ref guide building...
{noformat}
[INFO] Generating "Dependency Management" report --- maven-project-info-reports-plugin:3.0.0:dependency-management
[WARNING] Unable to create Maven project for org.xerial.snappy:snappy-java:jar:1.1.8.4 from repository.
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[ERROR] Unknown packaging: bundle @ line 6, column 16
at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:207)
at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:342)
at org.apache.maven.report.projectinfo.dependencies.RepositoryUtils.getMavenProjectFromRepository (RepositoryUtils.java:125)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.getDependencyRow (DependencyManagementRenderer.java:253)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderDependenciesForScope (DependencyManagementRenderer.java:202)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderDependenciesForAllScopes (DependencyManagementRenderer.java:151)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderSectionProjectDependencies (DependencyManagementRenderer.java:144)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderBody (DependencyManagementRenderer.java:130)
at org.apache.maven.reporting.AbstractMavenReportRenderer.render (AbstractMavenReportRenderer.java:80)
at org.apache.maven.report.projectinfo.DependencyManagementReport.executeReport (DependencyManagementReport.java:107)
at org.apache.maven.reporting.AbstractMavenReport.generate (AbstractMavenReport.java:251)
at org.apache.maven.plugins.site.render.ReportDocumentRenderer.renderDocument (ReportDocumentRenderer.java:230)
at org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.render (DefaultSiteRenderer.java:349)
at org.apache.maven.plugins.site.render.SiteMojo.renderLocale (SiteMojo.java:198)
at org.apache.maven.plugins.site.render.SiteMojo.execute (SiteMojo.java:147)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.model.building.ModelBuildingException: 1 problem was encountered while building the effective model for org.xerial.snappy:snappy-java:1.1.8.4
[ERROR] Unknown packaging: bundle @ line 6, column 16
at org.apache.maven.model.building.DefaultModelProblemCollector.newModelBuildingException (DefaultModelProblemCollector.java:197)
at org.apache.maven.model.building.DefaultModelBuilder.build (DefaultModelBuilder.java:498)
at org.apache.maven.model.building.DefaultModelBuilder.build (DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build (DefaultModelBuilder.java:430)
at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:173)
at org.apache.maven.project.DefaultProjectBuilder.build (DefaultProjectBuilder.java:342)
at org.apache.maven.report.projectinfo.dependencies.RepositoryUtils.getMavenProjectFromRepository (RepositoryUtils.java:125)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.getDependencyRow (DependencyManagementRenderer.java:253)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderDependenciesForScope (DependencyManagementRenderer.java:202)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderDependenciesForAllScopes (DependencyManagementRenderer.java:151)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderSectionProjectDependencies (DependencyManagementRenderer.java:144)
at org.apache.maven.report.projectinfo.dependencies.renderer.DependencyManagementRenderer.renderBody (DependencyManagementRenderer.java:130)
at org.apache.maven.reporting.AbstractMavenReportRenderer.render (AbstractMavenReportRenderer.java:80)
at org.apache.maven.report.projectinfo.DependencyManagementReport.executeReport (DependencyManagementReport.java:107)
at org.apache.maven.reporting.AbstractMavenReport.generate (AbstractMavenReport.java:251)
at org.apache.maven.plugins.site.render.ReportDocumentRenderer.renderDocument (ReportDocumentRenderer.java:230)
at org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.render (DefaultSiteRenderer.java:349)
at org.apache.maven.plugins.site.render.SiteMojo.renderLocale (SiteMojo.java:198)
at org.apache.maven.plugins.site.render.SiteMojo.execute (SiteMojo.java:147)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
{noformat}
No sure what's the problem but we use the same version of snappy-java on master, but master is OK...
> Fallback support to pure Java compression
> -----------------------------------------
>
> Key: HBASE-26259
> URL: https://issues.apache.org/jira/browse/HBASE-26259
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
> Attachments: BenchmarkCodec.java, BenchmarksMain.java, RandomDistribution.java, ac_lz4_results.pdf, ac_snappy_results.pdf, ac_zstd_results.pdf, lz4_lz4-java_result.pdf, xerial_snappy_results.pdf
>
>
> Airlift’s aircompressor (https://search.maven.org/artifact/io.airlift/aircompressor) is an Apache 2 licensed library, for Java 8 and up, available in Maven central, which provides pure Java implementations of gzip, lz4, lzo, snappy, and zstd and Hadoop compression codecs for same, claiming “_they are typically 300% faster than the JNI wrappers_.” (https://github.com/airlift/aircompressor). This library is under active development and up to date releases because it is used by Trino.
> Proposed changes:
> * Modify Compression.java such that compression codec implementation classes can be specified by configuration. Currently they are hardcoded as strings.
> * Pull in aircompressor as a ‘compile’ time dependency so it will be bundled into our build and made available on the server classpath.
> * Modify Compression.java to fall back to an aircompressor pure Java implementation if schema specifies a compression algorithm, a Hadoop native codec was specified as desired implementation, but the requisite native support is somehow not available.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)