You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Miguel Pereira <mi...@gmail.com> on 2013/11/06 22:09:48 UTC
HDFS durable sync
Hia guys,
Whats the recommended setting for HDFS durable sync CDH4.4 or how do I find
out?
....
Cloudera CDH 3u0-3u3 ???? true
Cloudera CDH 3u4 dfs.support.append true
....
Cheers,
Miguel
Re: HDFS durable sync
Posted by Josh Elser <jo...@gmail.com>.
It's just the README that's incomplete.
I missed it in my update of
https://issues.apache.org/jira/browse/ACCUMULO-1637.
On 11/6/13, 4:39 PM, John Vines wrote:
> Does CDH default to that the config file or just the hardcoded default? If
> it's the latter, it still needs to be put in because older versions had no
> single place to look in to determine defaults so tservers will fail to
> start if it cannot find one of those settings.
>
> On Wed, Nov 6, 2013 at 4:34 PM, Joey Echeverria <jo...@clouderagovt.com>wrote:
>
>> The setting is the same as the one for Apache Hadoop 2.0.5-alpha and
>> 2.2.0 (dfs.support.append), but it defaults to true in CDH4 anyway, so
>> you shouldn't have to change it.
>>
>> If you have more CDH specific questions, you can use the CDH mailing
>> list or public forum. Ping me off-list if you need a pointer to the
>> location of either resource.
>>
>> -Joey
>>
>> On Wed, Nov 6, 2013 at 4:09 PM, Miguel Pereira
>> <mi...@gmail.com> wrote:
>>> Hia guys,
>>>
>>> Whats the recommended setting for HDFS durable sync CDH4.4 or how do I
>> find
>>> out?
>>> ....
>>>
>>> Cloudera CDH 3u0-3u3 ???? true
>>> Cloudera CDH 3u4 dfs.support.append true
>>>
>>> ....
>>>
>>> Cheers,
>>> Miguel
>>
>
Re: HDFS durable sync
Posted by Miguel Pereira <mi...@gmail.com>.
Thanks Joey, Ill try that tomorrow. and report logs/findings.
Re: HDFS durable sync
Posted by Eric Newton <er...@gmail.com>.
It's related, in that it reduces data loss, but it's a different setting.
We're constantly syncing the WALog, but not data in RFiles. To get
datanodes to sync when a file is closed, you should use
dfs.datanode.synconclose=true.
-Eric
On Fri, Nov 8, 2013 at 3:32 PM, Miguel Pereira <mi...@gmail.com>wrote:
> Alright guys, with Accumulo 1.5.0 and CDH4.4 we are getting this warning:
>
> 2013-11-08 15:11:02,672 [tabletserver.TabletServer] WARN :
> > dfs.datanode.synconclose set to false: data loss is possible on system
> > reset or power loss
> >
>
> I was wondering if this is related to the durable sync and the recommended
> way to change the config settings ( xml files, Cloudera manager ).
>
> Cheers,
> Miguel
>
Re: HDFS durable sync
Posted by Miguel Pereira <mi...@gmail.com>.
Alright guys, with Accumulo 1.5.0 and CDH4.4 we are getting this warning:
2013-11-08 15:11:02,672 [tabletserver.TabletServer] WARN :
> dfs.datanode.synconclose set to false: data loss is possible on system
> reset or power loss
>
I was wondering if this is related to the durable sync and the recommended
way to change the config settings ( xml files, Cloudera manager ).
Cheers,
Miguel
Re: HDFS durable sync
Posted by Joey Echeverria <jo...@clouderagovt.com>.
It's set in the hdfs-default.xml that comes in the
hadoop-hdfs-<version>.jar. So if the tserver picks it up by creating a
Configuration() object, it should see it.
-Joey
On Wed, Nov 6, 2013 at 4:39 PM, John Vines <vi...@apache.org> wrote:
> Does CDH default to that the config file or just the hardcoded default? If
> it's the latter, it still needs to be put in because older versions had no
> single place to look in to determine defaults so tservers will fail to
> start if it cannot find one of those settings.
>
> On Wed, Nov 6, 2013 at 4:34 PM, Joey Echeverria <jo...@clouderagovt.com>wrote:
>
>> The setting is the same as the one for Apache Hadoop 2.0.5-alpha and
>> 2.2.0 (dfs.support.append), but it defaults to true in CDH4 anyway, so
>> you shouldn't have to change it.
>>
>> If you have more CDH specific questions, you can use the CDH mailing
>> list or public forum. Ping me off-list if you need a pointer to the
>> location of either resource.
>>
>> -Joey
>>
>> On Wed, Nov 6, 2013 at 4:09 PM, Miguel Pereira
>> <mi...@gmail.com> wrote:
>> > Hia guys,
>> >
>> > Whats the recommended setting for HDFS durable sync CDH4.4 or how do I
>> find
>> > out?
>> > ....
>> >
>> > Cloudera CDH 3u0-3u3 ???? true
>> > Cloudera CDH 3u4 dfs.support.append true
>> >
>> > ....
>> >
>> > Cheers,
>> > Miguel
>>
Re: HDFS durable sync
Posted by John Vines <vi...@apache.org>.
Does CDH default to that the config file or just the hardcoded default? If
it's the latter, it still needs to be put in because older versions had no
single place to look in to determine defaults so tservers will fail to
start if it cannot find one of those settings.
On Wed, Nov 6, 2013 at 4:34 PM, Joey Echeverria <jo...@clouderagovt.com>wrote:
> The setting is the same as the one for Apache Hadoop 2.0.5-alpha and
> 2.2.0 (dfs.support.append), but it defaults to true in CDH4 anyway, so
> you shouldn't have to change it.
>
> If you have more CDH specific questions, you can use the CDH mailing
> list or public forum. Ping me off-list if you need a pointer to the
> location of either resource.
>
> -Joey
>
> On Wed, Nov 6, 2013 at 4:09 PM, Miguel Pereira
> <mi...@gmail.com> wrote:
> > Hia guys,
> >
> > Whats the recommended setting for HDFS durable sync CDH4.4 or how do I
> find
> > out?
> > ....
> >
> > Cloudera CDH 3u0-3u3 ???? true
> > Cloudera CDH 3u4 dfs.support.append true
> >
> > ....
> >
> > Cheers,
> > Miguel
>
Re: HDFS durable sync
Posted by Joey Echeverria <jo...@clouderagovt.com>.
The setting is the same as the one for Apache Hadoop 2.0.5-alpha and
2.2.0 (dfs.support.append), but it defaults to true in CDH4 anyway, so
you shouldn't have to change it.
If you have more CDH specific questions, you can use the CDH mailing
list or public forum. Ping me off-list if you need a pointer to the
location of either resource.
-Joey
On Wed, Nov 6, 2013 at 4:09 PM, Miguel Pereira
<mi...@gmail.com> wrote:
> Hia guys,
>
> Whats the recommended setting for HDFS durable sync CDH4.4 or how do I find
> out?
> ....
>
> Cloudera CDH 3u0-3u3 ???? true
> Cloudera CDH 3u4 dfs.support.append true
>
> ....
>
> Cheers,
> Miguel