Accumulo init.d script

[Maria Krommyda <ma...@yahoo.gr>]

Hello everyone, 
I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop (2.2.0) and Accumulo (1.7.3)I am trying to set up a script at init.d that would start all three services in case of an unexpected reboot of the machine. 
I already have a start_me script that I use to start the services which works great, but making it an init.d script has been proven a challenge.
I have set all the environmental variables in /etc/default/my_script, which I include at the beginning of my script I run all the commands as su accumulo_user -c 'command', trying to avoid permission errors.
When I reboot the machine, I see the three services running (using the top command), I can access the Hadoop admin page but Accumulo gives as Error:"Connection refused at localhost:9000" and the monitor system doesn't start. 
Any ideas of what might be the problem, where I should look for any other log that might help will be highly appreciated. 
And if you have any thoughts about a relevant path that I might not have changed as fixed, please do share.
Thank you in advance for your time.
Best regards, Maria. 

Re: Accumulo init.d script

[Mike Miller <mm...@apache.org>]

Off the top of my head, I am not sure what to do for your problem.  But we
have released native packages for Fedora here:
https://src.fedoraproject.org/rpms/accumulo/tree/master
Perhaps the work we did to support those packages might help.

On Wed, Aug 8, 2018 at 3:32 AM Maria Krommyda <ma...@yahoo.gr> wrote:

> Hello everyone,
>
> I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop
> (2.2.0) and Accumulo (1.7.3)
> I am trying to set up a script at init.d that would start all three
> services in case of an unexpected reboot of the machine.
>
> I already have a start_me script that I use to start the services which
> works great, but making it an init.d script has been proven a challenge.
>
> I have set all the environmental variables in /etc/default/my_script,
> which I include at the beginning of my script
> I run all the commands as su accumulo_user -c 'command', trying to avoid
> permission errors.
>
> When I reboot the machine, I see the three services running (using the top
> command), I can access the Hadoop admin page but Accumulo gives as Error:
> "Connection refused at localhost:9000" and the monitor system doesn't
> start.
>
> Any ideas of what might be the problem, where I should look for any other
> log that might help will be highly appreciated.
>
> And if you have any thoughts about a relevant path that I might not have
> changed as fixed, please do share.
>
> Thank you in advance for your time.
>
> Best regards,
> Maria.
>
>

Σχετ: Accumulo init.d script

[Maria Krommyda <ma...@yahoo.gr>]

Hello everyone, 
Thank you for your tips and advice. I will try everything out and get back to you regarding any new developments. 
@ Christopher: You are right in the sense that what you mentioned will not help me currently with solving my problem but making my own custom start script is something that I would have never thought and even though I am currently too press with time, I might give it a shot at one point. 
Thank you again for your time.
Maria.  

    Στις 7:33 μ.μ. Πέμπτη, 9 Αυγούστου 2018, ο/η Christopher <ct...@apache.org> έγραψε:
 

 In general, I've found Accumulo's launch scripts for the tarball distribution to be unsuitable for automating with system init stuffs, due to their complexity and reliance on various environment pieces (though that's not to say others haven't had a good experience using them).

For Fedora's RPM packaging (vs. the binary tarball), I opted to rewrite Accumulo's launch scripts with a standard "JPackage"-style generated launch script (a simple script which sources an env script, sets up class path, locates Java, and launches the main class with the provided args, all in a very simple way), and I called that script in very simple systemd scripts.

Perhaps rewriting the launch scripts isn't the best idea for your situation... I don't know... but if you find that Accumulo's default scripts are too problematic, it may be something to consider. I know one of the main benefits I saw was that I didn't have to bother with any of the complicated log capturing output-redirection stuff... I just let the process log to STDOUT with a simple log4j ConsoleAppender, and let systemd/journald handle the log management with its system-wide log management policy. I also didn't have to do any PID file management or complicated environment setup. If you write custom scripts tailored to your requirements, you may be able to see similar benefits.

The good news is that we've tried to take steps to ensure we don't rely on our provided scripts too heavily, and that processes can be run directly with your own scripts, by simply calling the main method with an appropriate classpath and args. We've taken some steps in existing releases to help decouple the code from the scripts, and we've done even more in that regard for 2.0.

I realize none of this helps you troubleshoot your existing init scripts... and I'm sorry for that... but I think it's worth noting that Accumulo services aren't too tightly coupled to the launch scripts, so if they can't work for you, it's an option to use your own.

As for possible things to look for troubleshooting your existing scripts (in addition to the other steps people have suggested):

1. Consider disabling SELinux to see if that changes anything. Accumulo services may use network resources disabled by default in your SELinux policy.
2. If you're using SysVInit scripts (init.d) instead of systemd, use existing functions from /etc/init.d/functions in your init script whenever possible... it can save you lots of headaches.
3. Avoid bash-isms; init.d scripts should be POSIX, and your scripts may behave differently at bootup than when run manually for this reason.
4. Use shellcheck to check for problems.
5. Check file permissions of all scripts and paths to scripts to ensure they are executable/readable/traversable by the root user and the Accumulo user.


On Thu, Aug 9, 2018 at 9:49 AM Keith Turner <ke...@deenlo.com> wrote:

I would try adding set -x to the Accumuo script you are running and
see what that outputs.  Could add that as the second line of a script
as follows.

 #! /usr/bin/env bash
set -x

Hopefully that will shed some light on the problem.

On Thu, Aug 9, 2018 at 7:20 AM, Maria Krommyda <ma...@yahoo.gr> wrote:
> Hello Josh,
>
> Thank you for your time and suggestions.
>
> I am setting the ACCUMULO_HOME variable and I see the logs in the proper
> directory being updated on every reboot so this should not be the problem.
>
> I do not get any error/warning is the syslog. Only the confirmation that the
> service was started. If there is any other log that I can check please let
> me know.
>
> What is leading me to believe that it is a path/permission/variable issue,
> is that the exact same script runs without any problem from the home
> directory after the login.
>
> Best regards,
> Maria.
>
>
>
> Στις 6:06 μ.μ. Τετάρτη, 8 Αυγούστου 2018, ο/η Josh Elser <el...@apache.org>
> έγραψε:
>
>
> Every Accumulo service creates log files in the directory you specified
> via the ACCUMULO_LOG_DIR environment variable in accumulo-env.sh
>
> If you didn't define this, it likely defaults to ACCUMULO_HOME/logs.
>
> Have you looked at your syslog or similar to understand what your init.d
> script's output was? If Hadoop comes up correctly and you know your
> steps work otherwise, it sounds like it might be a typo in your script.
>
> On 8/8/18 3:32 AM, Maria Krommyda wrote:
>> Hello everyone,
>>
>> I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop
>> (2.2.0) and Accumulo (1.7.3)
>> I am trying to set up a script at init.d that would start all three
>> services in case of an unexpected reboot of the machine.
>>
>> I already have a start_me script that I use to start the services which
>> works great, but making it an init.d script has been proven a challenge.
>>
>> I have set all the environmental variables in /etc/default/my_script,
>> which I include at the beginning of my script
>> I run all the commands as su accumulo_user -c 'command', trying to avoid
>> permission errors.
>>
>> When I reboot the machine, I see the three services running (using the
>> top command), I can access the Hadoop admin page but Accumulo gives as
>> Error:
>> "Connection refused at localhost:9000" and the monitor system doesn't
>> start.
>>
>> Any ideas of what might be the problem, where I should look for any
>> other log that might help will be highly appreciated.
>>
>> And if you have any thoughts about a relevant path that I might not have
>> changed as fixed, please do share.
>>
>> Thank you in advance for your time.
>>
>> Best regards,
>> Maria.
>>
>
>



   

Re: Accumulo init.d script

[Christopher <ct...@apache.org>]

In general, I've found Accumulo's launch scripts for the tarball
distribution to be unsuitable for automating with system init stuffs, due
to their complexity and reliance on various environment pieces (though
that's not to say others haven't had a good experience using them).

For Fedora's RPM packaging (vs. the binary tarball), I opted to rewrite
Accumulo's launch scripts with a standard "JPackage"-style generated launch
script (a simple script which sources an env script, sets up class path,
locates Java, and launches the main class with the provided args, all in a
very simple way), and I called that script in very simple systemd scripts.

Perhaps rewriting the launch scripts isn't the best idea for your
situation... I don't know... but if you find that Accumulo's default
scripts are too problematic, it may be something to consider. I know one of
the main benefits I saw was that I didn't have to bother with any of the
complicated log capturing output-redirection stuff... I just let the
process log to STDOUT with a simple log4j ConsoleAppender, and let
systemd/journald handle the log management with its system-wide log
management policy. I also didn't have to do any PID file management or
complicated environment setup. If you write custom scripts tailored to your
requirements, you may be able to see similar benefits.

The good news is that we've tried to take steps to ensure we don't rely on
our provided scripts too heavily, and that processes can be run directly
with your own scripts, by simply calling the main method with an
appropriate classpath and args. We've taken some steps in existing releases
to help decouple the code from the scripts, and we've done even more in
that regard for 2.0.

I realize none of this helps you troubleshoot your existing init scripts...
and I'm sorry for that... but I think it's worth noting that Accumulo
services aren't too tightly coupled to the launch scripts, so if they can't
work for you, it's an option to use your own.

As for possible things to look for troubleshooting your existing scripts
(in addition to the other steps people have suggested):

1. Consider disabling SELinux to see if that changes anything. Accumulo
services may use network resources disabled by default in your SELinux
policy.
2. If you're using SysVInit scripts (init.d) instead of systemd, use
existing functions from /etc/init.d/functions in your init script whenever
possible... it can save you lots of headaches.
3. Avoid bash-isms; init.d scripts should be POSIX, and your scripts may
behave differently at bootup than when run manually for this reason.
4. Use shellcheck to check for problems.
5. Check file permissions of all scripts and paths to scripts to ensure
they are executable/readable/traversable by the root user and the Accumulo
user.


On Thu, Aug 9, 2018 at 9:49 AM Keith Turner <ke...@deenlo.com> wrote:

> I would try adding set -x to the Accumuo script you are running and
> see what that outputs.  Could add that as the second line of a script
> as follows.
>
>  #! /usr/bin/env bash
> set -x
>
> Hopefully that will shed some light on the problem.
>
> On Thu, Aug 9, 2018 at 7:20 AM, Maria Krommyda <ma...@yahoo.gr>
> wrote:
> > Hello Josh,
> >
> > Thank you for your time and suggestions.
> >
> > I am setting the ACCUMULO_HOME variable and I see the logs in the proper
> > directory being updated on every reboot so this should not be the
> problem.
> >
> > I do not get any error/warning is the syslog. Only the confirmation that
> the
> > service was started. If there is any other log that I can check please
> let
> > me know.
> >
> > What is leading me to believe that it is a path/permission/variable
> issue,
> > is that the exact same script runs without any problem from the home
> > directory after the login.
> >
> > Best regards,
> > Maria.
> >
> >
> >
> > Στις 6:06 μ.μ. Τετάρτη, 8 Αυγούστου 2018, ο/η Josh Elser <
> elserj@apache.org>
> > έγραψε:
> >
> >
> > Every Accumulo service creates log files in the directory you specified
> > via the ACCUMULO_LOG_DIR environment variable in accumulo-env.sh
> >
> > If you didn't define this, it likely defaults to ACCUMULO_HOME/logs.
> >
> > Have you looked at your syslog or similar to understand what your init.d
> > script's output was? If Hadoop comes up correctly and you know your
> > steps work otherwise, it sounds like it might be a typo in your script.
> >
> > On 8/8/18 3:32 AM, Maria Krommyda wrote:
> >> Hello everyone,
> >>
> >> I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop
> >> (2.2.0) and Accumulo (1.7.3)
> >> I am trying to set up a script at init.d that would start all three
> >> services in case of an unexpected reboot of the machine.
> >>
> >> I already have a start_me script that I use to start the services which
> >> works great, but making it an init.d script has been proven a challenge.
> >>
> >> I have set all the environmental variables in /etc/default/my_script,
> >> which I include at the beginning of my script
> >> I run all the commands as su accumulo_user -c 'command', trying to avoid
> >> permission errors.
> >>
> >> When I reboot the machine, I see the three services running (using the
> >> top command), I can access the Hadoop admin page but Accumulo gives as
> >> Error:
> >> "Connection refused at localhost:9000" and the monitor system doesn't
> >> start.
> >>
> >> Any ideas of what might be the problem, where I should look for any
> >> other log that might help will be highly appreciated.
> >>
> >> And if you have any thoughts about a relevant path that I might not have
> >> changed as fixed, please do share.
> >>
> >> Thank you in advance for your time.
> >>
> >> Best regards,
> >> Maria.
> >>
> >
> >
>

Re: Accumulo init.d script

[Keith Turner <ke...@deenlo.com>]

I would try adding set -x to the Accumuo script you are running and
see what that outputs.  Could add that as the second line of a script
as follows.

 #! /usr/bin/env bash
set -x

Hopefully that will shed some light on the problem.

On Thu, Aug 9, 2018 at 7:20 AM, Maria Krommyda <ma...@yahoo.gr> wrote:
> Hello Josh,
>
> Thank you for your time and suggestions.
>
> I am setting the ACCUMULO_HOME variable and I see the logs in the proper
> directory being updated on every reboot so this should not be the problem.
>
> I do not get any error/warning is the syslog. Only the confirmation that the
> service was started. If there is any other log that I can check please let
> me know.
>
> What is leading me to believe that it is a path/permission/variable issue,
> is that the exact same script runs without any problem from the home
> directory after the login.
>
> Best regards,
> Maria.
>
>
>
> Στις 6:06 μ.μ. Τετάρτη, 8 Αυγούστου 2018, ο/η Josh Elser <el...@apache.org>
> έγραψε:
>
>
> Every Accumulo service creates log files in the directory you specified
> via the ACCUMULO_LOG_DIR environment variable in accumulo-env.sh
>
> If you didn't define this, it likely defaults to ACCUMULO_HOME/logs.
>
> Have you looked at your syslog or similar to understand what your init.d
> script's output was? If Hadoop comes up correctly and you know your
> steps work otherwise, it sounds like it might be a typo in your script.
>
> On 8/8/18 3:32 AM, Maria Krommyda wrote:
>> Hello everyone,
>>
>> I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop
>> (2.2.0) and Accumulo (1.7.3)
>> I am trying to set up a script at init.d that would start all three
>> services in case of an unexpected reboot of the machine.
>>
>> I already have a start_me script that I use to start the services which
>> works great, but making it an init.d script has been proven a challenge.
>>
>> I have set all the environmental variables in /etc/default/my_script,
>> which I include at the beginning of my script
>> I run all the commands as su accumulo_user -c 'command', trying to avoid
>> permission errors.
>>
>> When I reboot the machine, I see the three services running (using the
>> top command), I can access the Hadoop admin page but Accumulo gives as
>> Error:
>> "Connection refused at localhost:9000" and the monitor system doesn't
>> start.
>>
>> Any ideas of what might be the problem, where I should look for any
>> other log that might help will be highly appreciated.
>>
>> And if you have any thoughts about a relevant path that I might not have
>> changed as fixed, please do share.
>>
>> Thank you in advance for your time.
>>
>> Best regards,
>> Maria.
>>
>
>

Re: Σχετ: Accumulo init.d script

[Josh Elser <el...@apache.org>]

Look through all log files in ACCUMULO_LOG_DIR that have a timestamp 
later than when you restarted the VM. If none of the files for a 
specific Accumulo service have a newer timestamp, then it's a problem 
solely in your init.d script (as the service didn't get started correctly).

On 8/9/18 7:20 AM, Maria Krommyda wrote:
> Hello Josh,
> 
> Thank you for your time and suggestions.
> 
> I am setting the ACCUMULO_HOME variable and I see the logs in the proper 
> directory being updated on every reboot so this should not be the problem.
> 
> I do not get any error/warning is the syslog. Only the confirmation that 
> the service was started. If there is any other log that I can check 
> please let me know.
> 
> What is leading me to believe that it is a path/permission/variable 
> issue, is that the exact same script runs without any problem from the 
> home directory after the login.
> 
> Best regards,
> Maria.
> 
> 
> 
> Στις 6:06 μ.μ. Τετάρτη, 8 Αυγούστου 2018, ο/η Josh Elser 
> <el...@apache.org> έγραψε:
> 
> 
> Every Accumulo service creates log files in the directory you specified
> via the ACCUMULO_LOG_DIR environment variable in accumulo-env.sh
> 
> If you didn't define this, it likely defaults to ACCUMULO_HOME/logs.
> 
> Have you looked at your syslog or similar to understand what your init.d
> script's output was? If Hadoop comes up correctly and you know your
> steps work otherwise, it sounds like it might be a typo in your script.
> 
> On 8/8/18 3:32 AM, Maria Krommyda wrote:
>  > Hello everyone,
>  >
>  > I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop
>  > (2.2.0) and Accumulo (1.7.3)
>  > I am trying to set up a script at init.d that would start all three
>  > services in case of an unexpected reboot of the machine.
>  >
>  > I already have a start_me script that I use to start the services which
>  > works great, but making it an init.d script has been proven a challenge.
>  >
>  > I have set all the environmental variables in /etc/default/my_script,
>  > which I include at the beginning of my script
>  > I run all the commands as su accumulo_user -c 'command', trying to avoid
>  > permission errors.
>  >
>  > When I reboot the machine, I see the three services running (using the
>  > top command), I can access the Hadoop admin page but Accumulo gives as
>  > Error:
>  > "Connection refused at localhost:9000" and the monitor system doesn't
>  > start.
>  >
>  > Any ideas of what might be the problem, where I should look for any
>  > other log that might help will be highly appreciated.
>  >
>  > And if you have any thoughts about a relevant path that I might not have
>  > changed as fixed, please do share.
>  >
>  > Thank you in advance for your time.
>  >
>  > Best regards,
>  > Maria.
>  >
> 
> 

Σχετ: Accumulo init.d script

[Maria Krommyda <ma...@yahoo.gr>]

Hello Josh, 
Thank you for your time and suggestions.
I am setting the ACCUMULO_HOME variable and I see the logs in the proper directory being updated on every reboot so this should not be the problem.
I do not get any error/warning is the syslog. Only the confirmation that the service was started. If there is any other log that I can check please let me know.
What is leading me to believe that it is a path/permission/variable issue, is that the exact same script runs without any problem from the home directory after the login.
Best regards, Maria. 


    Στις 6:06 μ.μ. Τετάρτη, 8 Αυγούστου 2018, ο/η Josh Elser <el...@apache.org> έγραψε:
 

 Every Accumulo service creates log files in the directory you specified 
via the ACCUMULO_LOG_DIR environment variable in accumulo-env.sh

If you didn't define this, it likely defaults to ACCUMULO_HOME/logs.

Have you looked at your syslog or similar to understand what your init.d 
script's output was? If Hadoop comes up correctly and you know your 
steps work otherwise, it sounds like it might be a typo in your script.

On 8/8/18 3:32 AM, Maria Krommyda wrote:
> Hello everyone,
> 
> I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop 
> (2.2.0) and Accumulo (1.7.3)
> I am trying to set up a script at init.d that would start all three 
> services in case of an unexpected reboot of the machine.
> 
> I already have a start_me script that I use to start the services which 
> works great, but making it an init.d script has been proven a challenge.
> 
> I have set all the environmental variables in /etc/default/my_script, 
> which I include at the beginning of my script
> I run all the commands as su accumulo_user -c 'command', trying to avoid 
> permission errors.
> 
> When I reboot the machine, I see the three services running (using the 
> top command), I can access the Hadoop admin page but Accumulo gives as 
> Error:
> "Connection refused at localhost:9000" and the monitor system doesn't 
> start.
> 
> Any ideas of what might be the problem, where I should look for any 
> other log that might help will be highly appreciated.
> 
> And if you have any thoughts about a relevant path that I might not have 
> changed as fixed, please do share.
> 
> Thank you in advance for your time.
> 
> Best regards,
> Maria.
> 


   

Re: Accumulo init.d script

[Josh Elser <el...@apache.org>]

Every Accumulo service creates log files in the directory you specified 
via the ACCUMULO_LOG_DIR environment variable in accumulo-env.sh

If you didn't define this, it likely defaults to ACCUMULO_HOME/logs.

Have you looked at your syslog or similar to understand what your init.d 
script's output was? If Hadoop comes up correctly and you know your 
steps work otherwise, it sounds like it might be a typo in your script.

On 8/8/18 3:32 AM, Maria Krommyda wrote:
> Hello everyone,
> 
> I have set up a VM, Ubuntu 16.04, where I run Zookeeper (3.4.6), Hadoop 
> (2.2.0) and Accumulo (1.7.3)
> I am trying to set up a script at init.d that would start all three 
> services in case of an unexpected reboot of the machine.
> 
> I already have a start_me script that I use to start the services which 
> works great, but making it an init.d script has been proven a challenge.
> 
> I have set all the environmental variables in /etc/default/my_script, 
> which I include at the beginning of my script
> I run all the commands as su accumulo_user -c 'command', trying to avoid 
> permission errors.
> 
> When I reboot the machine, I see the three services running (using the 
> top command), I can access the Hadoop admin page but Accumulo gives as 
> Error:
> "Connection refused at localhost:9000" and the monitor system doesn't 
> start.
> 
> Any ideas of what might be the problem, where I should look for any 
> other log that might help will be highly appreciated.
> 
> And if you have any thoughts about a relevant path that I might not have 
> changed as fixed, please do share.
> 
> Thank you in advance for your time.
> 
> Best regards,
> Maria.
>