Using flume and a Google Cloud Storage (GCS) Sink.

["Guyle M. Taber" <gu...@gmtech.net>]

I’m having trouble getting a flume sink to GCS working. There’s not a lot of documentation out there for GCS and Flume.

Does anyone have a GCS sink in Flume working?

I can successfully list the contents of the GCS bucket using “gustils” and the values in core-site-xml seem valid, as I can list the contents of the bucket using "hdfs dfs -ls gs://m <gs://m>y-bucket”.  This flume sink when configured for an avro sink to HDFS works perfectly, but pointing to a GCS bucket, no data is sent and flume shows no errors.

Here’s my GCS sink in flume.conf

agent.sinks.HDFSSink.type = hdfs
agent.sinks.HDFSSink.hdfs.path = gs://my-bucket/
agent.sinks.HDFSSink.channel = memoryChannel
agent.sinks.HDFSSink.hdfs.useLocalTimeStamp = true
agent.sinks.HDFSSink.hdfs.rollCount = 0
agent.sinks.HDFSSink.hdfs.rollInterval = 0
agent.sinks.HDFSSink.hdfs.rollSize = 0
agent.sinks.HDFSSink.hdfs.fileType = DataStream
agent.sinks.HDFSSink.hdfs.writeFormat = Text
agent.sinks.HDFSSink.hdfs.filePrefix = %H-%M-%S
agent.sinks.HDFSSink.hdfs.filePrefix = .json