You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by chandra sekar <th...@hotmail.com> on 2016/05/10 03:03:51 UTC
Twitter Source problem
Hi I am using Twitter source to collect the data from twitter. Suddenly the process throw an error and there is no data from twitter feed.
Error:
404:The URI requested is invalid or the resource requested, such as a user, does not exist.
Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api
Configuration:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
# TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey =
TwitterAgent.sources.Twitter.consumerSecret =
TwitterAgent.sources.Twitter.accessToken =
TwitterAgent.sources.Twitter.accessTokenSecret =
TwitterAgent.sources.Twitter.keywords = data analytics, data science , hadoop , big data
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/data_analytics
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 100000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 500000
TwitterAgent.channels.MemChannel.transactionCapacity = 3000
Thanks & Regards
Chandrasekar
Re: Twitter Source problem
Posted by Ronald Van de Kuil <ro...@nl.ibm.com>.
Hello,
I used the following class:
a1.sources.r1.type = com.cloudera.flume.source.TwitterSource
It is running stable since Februari 2016.
Hope that helps.
Met Vriendelijke Groet,
Ronald van de Kuil
From: chandra sekar <th...@hotmail.com>
To: "user@flume.apache.org" <us...@flume.apache.org>
Date: 12-05-16 03:38
Subject: Re: Twitter Source problem
Dear Ronald,
I have a twitter apps account and created all the keys. here I did removed
the keys due to the security purpose.
Regards
Chandrasekar.
Sent from Outlook
From: Ronald Van de Kuil <ro...@nl.ibm.com>
Sent: Tuesday, May 10, 2016 2:10 PM
To: user@flume.apache.org
Subject: Re: Twitter Source problem
You would have to create a twitter app (account) to get your key, token
and so on.
Op 10 mei 2016 om 05:04 heeft chandra sekar <th...@hotmail.com> het
volgende geschreven:
Hi I am using Twitter source to collect the data from twitter. Suddenly
the process throw an error and there is no data from twitter feed.
Error:
404:The URI requested is invalid or the resource requested, such as a
user, does not exist.
Unknown URL. See Twitter Streaming API documentation at
http://dev.twitter.com/pages/streaming_api
Configuration:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type =
com.cloudera.flume.source.TwitterSource
# TwitterAgent.sources.Twitter.type =
org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey =
TwitterAgent.sources.Twitter.consumerSecret =
TwitterAgent.sources.Twitter.accessToken =
TwitterAgent.sources.Twitter.accessTokenSecret =
TwitterAgent.sources.Twitter.keywords = data analytics, data science ,
hadoop , big data
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/data_analytics
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 100000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 500000
TwitterAgent.channels.MemChannel.transactionCapacity = 3000
Thanks & Regards
Chandrasekar
Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214
Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214
Re: Twitter Source problem
Posted by chandra sekar <th...@hotmail.com>.
Dear Ronald,
I have a twitter apps account and created all the keys. here I did removed the keys due to the security purpose.
Regards
Chandrasekar.
Sent from Outlook<http://aka.ms/weboutlook>
________________________________
From: Ronald Van de Kuil <ro...@nl.ibm.com>
Sent: Tuesday, May 10, 2016 2:10 PM
To: user@flume.apache.org
Subject: Re: Twitter Source problem
You would have to create a twitter app (account) to get your key, token and so on.
Op 10 mei 2016 om 05:04 heeft chandra sekar <th...@hotmail.com>> het volgende geschreven:
Hi I am using Twitter source to collect the data from twitter. Suddenly the process throw an error and there is no data from twitter feed.
Error:
404:The URI requested is invalid or the resource requested, such as a user, does not exist.
Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api
Configuration:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
# TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey =
TwitterAgent.sources.Twitter.consumerSecret =
TwitterAgent.sources.Twitter.accessToken =
TwitterAgent.sources.Twitter.accessTokenSecret =
TwitterAgent.sources.Twitter.keywords = data analytics, data science , hadoop , big data
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/data_analytics
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 100000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 500000
TwitterAgent.channels.MemChannel.transactionCapacity = 3000
Thanks & Regards
Chandrasekar
Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214
Re: Twitter Source problem
Posted by Ronald Van de Kuil <ro...@nl.ibm.com>.
You would have to create a twitter app (account) to get your key, token and so on.
> Op 10 mei 2016 om 05:04 heeft chandra sekar <th...@hotmail.com> het volgende geschreven:
>
> Hi I am using Twitter source to collect the data from twitter. Suddenly the process throw an error and there is no data from twitter feed.
>
> Error:
>
> 404:The URI requested is invalid or the resource requested, such as a user, does not exist.
> Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api
>
> Configuration:
>
> TwitterAgent.sources = Twitter
> TwitterAgent.channels = MemChannel
> TwitterAgent.sinks = HDFS
>
> TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
> # TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
> TwitterAgent.sources.Twitter.channels = MemChannel
> TwitterAgent.sources.Twitter.consumerKey =
> TwitterAgent.sources.Twitter.consumerSecret =
> TwitterAgent.sources.Twitter.accessToken =
> TwitterAgent.sources.Twitter.accessTokenSecret =
>
> TwitterAgent.sources.Twitter.keywords = data analytics, data science , hadoop , big data
>
> TwitterAgent.sinks.HDFS.channel = MemChannel
> TwitterAgent.sinks.HDFS.type = hdfs
> TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/data_analytics
> TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
> TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
> TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
> TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
> TwitterAgent.sinks.HDFS.hdfs.rollCount = 100000
>
> TwitterAgent.channels.MemChannel.type = memory
> TwitterAgent.channels.MemChannel.capacity = 500000
> TwitterAgent.channels.MemChannel.transactionCapacity = 3000
>
>
> Thanks & Regards
> Chandrasekar
Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214