You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "Deepak Garg (Jira)" <ji...@apache.org> on 2023/05/12 10:25:00 UTC
[jira] [Commented] (FLUME-3475) Upgrade Commons-dbcp 1.4 to Commons-dbcp2 2.9.0
[ https://issues.apache.org/jira/browse/FLUME-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722106#comment-17722106 ]
Deepak Garg commented on FLUME-3475:
------------------------------------
Logs after Flume started with jdbc-channel with persistent storage as DERBY.
Currently Flume only supports DERBY.
{code:java}
11 May 2023 14:31:43,990 INFO [main] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:154) - Creating channels
11 May 2023 14:31:43,993 INFO [main] (org.apache.flume.channel.DefaultChannelFactory.create:42) - Creating instance of channel jdbc_channel type jdbc
11 May 2023 14:31:43,997 WARN [main] (org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.initializeDataSource:389) - No connection URL specified. Using embedded derby database instance.
11 May 2023 14:31:43,998 WARN [main] (org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.initializeDataSource:424) - Overriding values for - driver: org.apache.derby.jdbc.EmbeddedDriver, user: saconnectUrl: jdbc:derby:/root/.flume/jdbc-channel/db;create=true, jdbc properties file: null, dbtype: DERBY
11 May 2023 14:31:45,170 WARN [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.schemaExists:376) - Schema for FLUME does not exist
11 May 2023 14:31:45,175 INFO [main] (org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.initializeSchema:221) - Foreign Key Constratins will be disabled.
11 May 2023 14:31:45,184 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE SCHEMA FLUME) Update count: 0
11 May 2023 14:31:45,238 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE TABLE FLUME.FL_EVENT ( FLE_ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, FLE_PAYLOAD VARCHAR(16384) FOR BIT DATA, FLE_CHANNEL VARCHAR(64), FLE_SPILL BOOLEAN)) Update count: 0
11 May 2023 14:31:45,249 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE TABLE FLUME.FL_PLSPILL ( FLP_EVENT BIGINT, FLP_SPILL BLOB)) Update count: 0
11 May 2023 14:31:45,268 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE TABLE FLUME.FL_HEADER ( FLH_ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, FLH_EVENT BIGINT, FLH_NAME VARCHAR(251), FLH_VALUE VARCHAR(251), FLH_NMSPILL BOOLEAN, FLH_VLSPILL BOOLEAN)) Update count: 0
11 May 2023 14:31:45,275 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE TABLE FLUME.FL_NMSPILL ( FLN_HEADER BIGINT, FLN_SPILL VARCHAR(32517))) Update count: 0
11 May 2023 14:31:45,280 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE TABLE FLUME.FL_VLSPILL ( FLV_HEADER BIGINT, FLV_SPILL VARCHAR(32517))) Update count: 0
11 May 2023 14:31:45,291 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE INDEX FLUME.IDX_FLE_CHANNEL ON FLUME.FL_EVENT (FLE_CHANNEL)) Update count: 0
11 May 2023 14:31:45,300 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE INDEX FLUME.IDX_FLH_EVENT ON FLUME.FL_HEADER (FLH_EVENT)) Update count: 0
11 May 2023 14:31:45,308 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE INDEX FLUME.IDX_FLP_EVENT ON FLUME.FL_PLSPILL (FLP_EVENT)) Update count: 0
11 May 2023 14:31:45,317 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE INDEX FLUME.IDX_FLN_HEADER ON FLUME.FL_NMSPILL (FLN_HEADER)) Update count: 0
11 May 2023 14:31:45,326 INFO [main] (org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.runQuery:547) - QUERY(CREATE INDEX FLUME.IDX_FLV_HEADER ON FLUME.FL_VLSPILL (FLV_HEADER)) Update count: 0
11 May 2023 14:31:45,364 WARN [main] (org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.initializeChannelState:151) - JDBC channel will operate without a capacity limit.
11 May 2023 14:31:45,364 INFO [main] (org.apache.flume.channel.jdbc.JdbcChannel.configure:91) - JDBC Channel initialized: jdbc_channel
11 May 2023 14:31:45,364 INFO [main] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:208) - Created channel jdbc_channel
11 May 2023 14:31:45,365 INFO [main] (org.apache.flume.source.DefaultSourceFactory.create:41) - Creating instance of source source1, type spooldir
11 May 2023 14:31:45,375 INFO [main] (org.apache.flume.sink.DefaultSinkFactory.create:42) - Creating instance of sink: sink1, type: hdfs
11 May 2023 14:31:45,383 INFO [main] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:123) - Channel jdbc_channel connected to [source1, sink1]
11 May 2023 14:31:45,383 INFO [main] (org.apache.flume.node.Application.initializeAllComponents:177) - Initializing components
11 May 2023 14:31:45,385 INFO [main] (org.apache.flume.node.Application.startAllComponents:207) - Starting new configuration:{ sourceRunners:{source1=EventDrivenSourceRunner: { source:Spool Directory source source1: { spoolDir: /home/deepgarg/flumetesting/spooldir } }} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@77b21474 counterGroup:{ name:null counters:{} } }} channels:{jdbc_channel=org.apache.flume.channel.jdbc.JdbcChannel{name: jdbc_channel}} }
11 May 2023 14:31:45,385 INFO [main] (org.apache.flume.node.Application.startAllComponents:214) - Starting Channel jdbc_channel
11 May 2023 14:31:45,386 INFO [main] (org.apache.flume.node.Application.startAllComponents:229) - Waiting for channel: jdbc_channel to start. Sleeping for 500 ms
11 May 2023 14:31:45,887 INFO [main] (org.apache.flume.node.Application.startAllComponents:241) - Starting Sink sink1
11 May 2023 14:31:45,887 INFO [main] (org.apache.flume.node.Application.startAllComponents:252) - Starting Source source1
11 May 2023 14:31:45,888 INFO [lifecycleSupervisor-1-1] (org.apache.flume.source.SpoolDirectorySource.start:85) - SpoolDirectorySource source starting with directory: /home/deepgarg/flumetesting/spooldir
11 May 2023 14:31:45,890 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:119) - Monitored counter group for type: SINK, name: sink1: Successfully registered new MBean.
11 May 2023 14:31:45,890 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:95) - Component type: SINK, name: sink1 started
11 May 2023 14:31:45,924 INFO [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:119) - Monitored counter group for type: SOURCE, name: source1: Successfully registered new MBean.
11 May 2023 14:31:45,924 INFO [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:95) - Component type: SOURCE, name: source1 started {code}
Logs after file has been created in HDFS
{code:java}
12 May 2023 06:17:13,001 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSDataStream.configure:57) - Serializer = TEXT, UseRawLocalFileSystem = false
12 May 2023 06:17:13,091 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.open:246) - Creating /tmp/thisisflume/_events.1683872233002.log.tmp
12 May 2023 06:17:44,239 INFO [hdfs-sink1-roll-timer-0] (org.apache.flume.sink.hdfs.HDFSEventSink$1.run:391) - Writer callback called.
12 May 2023 06:17:44,239 INFO [hdfs-sink1-roll-timer-0] (org.apache.flume.sink.hdfs.BucketWriter.doClose:438) - Closing /tmp/thisisflume/_events.1683872233002.log.tmp
12 May 2023 06:17:44,271 INFO [hdfs-sink1-call-runner-4] (org.apache.flume.sink.hdfs.BucketWriter$7.call:681) - Renaming /tmp/thisisflume/_events.1683872233002.log.tmp to /tmp/thisisflume/events.1683872233002.log {code}
output
{code:java}
[flumetesting]# hdfs dfs -cat /tmp/thisisflume/events.1683872233002.log
testing flume on 37 machine with jdbc-channel with derby {code}
Configuration used
{code:java}
agent2.channels=jdbc_channel
agent2.channels.jdbc_channel.type=jdbc
agent2.channels.jdbc_channel.db.type=DERBY
agent2.channels.jdbc_channel.driver.class=
agent2.channels.jdbc_channel.driver.url=
agent2.channels.jdbc_channel.db.username=sa
agent2.channels.jdbc_channel.db.password=
agent2.channels.jdbc_channel.create.schema=true
agent2.channels.jdbc_channel.create.index=true
agent2.channels.jdbc_channel.create.foreignkey=false
agent2.channels.jdbc_channel.transaction.isolation=READ_COMMITTED
agent2.channels.jdbc_channel.maximum.connections=10
agent2.channels.jdbc_channel.maximum.capacity=0
agent2.sinks=sink1
agent2.sinks.sink1.channel=jdbc_channel
agent2.sinks.sink1.type = hdfs
agent2.sinks.sink1.hdfs.path = /tmp/thisisflume
agent2.sinks.sink1.hdfs.filePrefix = events
agent2.sinks.sink1.hdfs.fileSuffix = .log
agent2.sinks.sink1.hdfs.inUsePrefix = _
agent2.sinks.sink1.hdfs.fileType = DataStream
agent2.sources=source1
agent2.sources.source1.channels=jdbc_channel
agent2.sources.source1.type=spooldir
agent2.sources.source1.spoolDir=/home/deepgarg/flumetesting/spooldir {code}
> Upgrade Commons-dbcp 1.4 to Commons-dbcp2 2.9.0
> ------------------------------------------------
>
> Key: FLUME-3475
> URL: https://issues.apache.org/jira/browse/FLUME-3475
> Project: Flume
> Issue Type: Dependency upgrade
> Environment: Hadoop 3
> RHEL 7
> flume 1.11.0
> Reporter: Deepak Garg
> Assignee: Deepak Garg
> Priority: Major
>
> This dependency upgrade requires changes in flume-jdbc-channel
>
> Explanation
> The Apache Commons DBCP packages are vulnerable to Insufficiently Protected Credentials. The {{toString}} method in various classes as mentioned below, displays sensitive credentials. An attacker can exploit this as part of a larger attack, using said credentials to gain unauthorized access.
> {_}Vulnerable Classes{_}:
> * DelegatingConnection.class
> * DriverConnectionFactory.class
> * DriverAdapterCPDS.class
> * PoolKey.class
> * UserPassKey.class
> Detection
> The application is vulnerable by using this component.
> Recommendation
> There is no non-vulnerable upgrade path for this component/package. We recommend investigating alternative components or a potential mitigating control.
> Version Affected
> [1.2.1,1.4]
> Root Cause
> commons-dbcp-1.4.jarorg/apache/commons/dbcp/DriverConnectionFactory.class[1.2.1, 20030818.201141)
> commons-dbcp-1.4.jarorg/apache/commons/dbcp/datasources/PoolKey.class[1.2.1, 20030818.201141)
> commons-dbcp-1.4.jarorg/apache/commons/dbcp/datasources/UserPassKey.class[1.2.1, 20030818.201141)
> commons-dbcp-1.4.jarorg/apache/commons/dbcp/DelegatingConnection.class[1.2.2, 20030818.201141)
> Advisories
> Project[https://github.com/apache/commons-dbcp/commit/a4c5af0da1de3a7f50c72fc7edaa1f653ca276dd]
> CVSS Details
> Sonatype CVSS 37.5
> CVSS VectorCVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org