You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by June Taylor <ju...@umn.edu> on 2016/06/29 17:58:36 UTC
Escaped characters in the 'env' argument passed to mesos-execute
Greetings,
We are using mesos-execute and passing an argument that contains a Unix
filepath, as follows:
*mesos-execute --master=cluster.example.com:5050
<http://cluster.example.com:5050> --name=test-program
--command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
"/home/john/temp_output/test.csv"}*
The slashes in the file path are then apparently escaped by mesos-execute
and therefore fail to be useful once picked up by the executor:
*FileNotFoundError: [Errno 2] No such file or directory:
'"\\/home\\/john\\/temp_output\\/test.csv"'*
*I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
Are other people experiencing this? What is the method to avoid it?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by June Taylor <ju...@umn.edu>.
Chris,
Thanks for the info - we do suspect it's abount mesos-execute, yes.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Thu, Jun 30, 2016 at 1:10 PM, Chris Baker <ch...@galacticfog.com> wrote:
> I tested with "/usr/bin/env && /usr/bin/sleep 30" to give myself a chance
> to look at stdout before the scheduler removed itself. I noticed that the
> problem is still in place; all environment variables have quotes and
> escaped slashes.
>
> This isn't a problem with launching a similar job via marathon:
> {
> "id": "/chris/test",
> "cmd": "/usr/bin/env && /usr/bin/sleep 30",
> "cpus": 0.1,
> "mem": 128,
> "disk": 0,
> "instances": 1,
> "env": {
> "MY_FILEPATH": "/home/john/temp_output/test.csv"
> }
> }
>
> stdout:
> Registered executor on 10.0.2.56
> Starting task chris_test.4fa0ac52-3eed-11e6-99b5-06094a33b9e2
> sh -c '/usr/bin/env && /usr/bin/sleep 30'
> Forked command at 7174
> ...
> MY_FILEPATH=/home/john/temp_output/test.csv
> ...
>
> So, I would assume it's something in mesos-exec. I haven't found the code
> that does it, so I can't figure out how to subvert it.
>
> On Thu, Jun 30, 2016 at 1:52 PM haosdent <ha...@gmail.com> wrote:
>
>> May you show the content of /home/john/mesos/error_msg.py I would like
>> to try it in my machine instead of executing `env` command.
>>
>> On Fri, Jul 1, 2016 at 1:40 AM, June Taylor <ju...@umn.edu> wrote:
>>
>>> Both of these suggestions are still resulting in this content being seen
>>> by the running application: \\/home\\/john\\/temp_output\\/test.csv"
>>>
>>> Thank you for your continued suggestions. Please try them out yourself
>>> and let me know if you can get a non-escaped path to print out successfully
>>> in your executing program, and let me know what you use to do that.
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker <ch...@galacticfog.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Thu, Jun 30, 2016 at 1:15 PM haosdent <ha...@gmail.com> wrote:
>>>>
>>>>> Actually you could write your json into a file and pass the file path
>>>>> into the flag. For example
>>>>>
>>>>> ```
>>>>> $ cat /tmp/a.json
>>>>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>> ```
>>>>>
>>>>> And use `file:///tmp/a.json` as the value of env flag.
>>>>>
>>>>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <ew...@groupon.com>
>>>>> wrote:
>>>>>
>>>>>> Heh, fair enough. I know for the master/agent daemons you can put
>>>>>> the config values into files, which makes this pretty clean (though
>>>>>> that probably depends on the OS/init-launcher-daemon you are using). But
>>>>>> yeah, when using the CLI you have to be pretty careful.
>>>>>>
>>>>>> - Erik
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On a side note.... requiring people to put JSON on the command line
>>>>>>> is a sadistic thing to do.
>>>>>>>
>>>>>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <
>>>>>>> eweathers@groupon.com> wrote:
>>>>>>>
>>>>>>>> +1 I would wrap every string in quotes... otherwise your shell
>>>>>>>> doesn't know what you mean. i.e.,
>>>>>>>>
>>>>>>>> How is the shell supposed to know that you want this all to be 1
>>>>>>>> string value for the --command parameter?
>>>>>>>>
>>>>>>>> --command=/home/john/anaconda3/bin/python
>>>>>>>> /home/john/mesos/error_msg.py read
>>>>>>>>
>>>>>>>> Similarly how is the shell to know that you *don't* want to end
>>>>>>>> this command and start a new one when you have a bare semicolon?
>>>>>>>>
>>>>>>>> --resources=cpus:1;mem:128
>>>>>>>>
>>>>>>>> - Erik
>>>>>>>>
>>>>>>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I use bash as well. How about
>>>>>>>>> ```
>>>>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>>>>>> --command="/home/john/anaconda3/bin/python
>>>>>>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>>>>
>>>>>>>>>> hasodent,
>>>>>>>>>>
>>>>>>>>>> We've tried your suggestion but it seems like that's not working
>>>>>>>>>> out:
>>>>>>>>>>
>>>>>>>>>> mesos-execute --master=cluster.example.com:5050
>>>>>>>>>> --name=test-program --command=/home/john/anaconda3/bin/python
>>>>>>>>>> /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128
>>>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>>>
>>>>>>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>>>>>>>>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error
>>>>>>>>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We're using the Bash shell. Is anyone else passing file-paths
>>>>>>>>>> within their environment variable argument?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> June Taylor
>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>> University of Minnesota
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <
>>>>>>>>>>> chris@galacticfog.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> It's your shell:
>>>>>>>>>>>>
>>>>>>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> hasodent,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I admit I don't see much of a difference in your version. Are
>>>>>>>>>>>>> you saying that Mesos doesn't escape the values, and that our program may
>>>>>>>>>>>>> be doing that on the executor end?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> June Taylor
>>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, @Taylor I use
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>>>>>>>>> --name="test"
>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Greetings,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We are using mesos-execute and passing an argument that
>>>>>>>>>>>>>>> contains a Unix filepath, as follows:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>>>>>>>>> executor:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>>>>>>>> shutdown*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Are other people experiencing this? What is the method to
>>>>>>>>>>>>>>> avoid it?
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> June Taylor
>>>>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Haosdent Huang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Haosdent Huang
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Haosdent Huang
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Haosdent Huang
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by Chris Baker <ch...@galacticfog.com>.
I tested with "/usr/bin/env && /usr/bin/sleep 30" to give myself a chance
to look at stdout before the scheduler removed itself. I noticed that the
problem is still in place; all environment variables have quotes and
escaped slashes.
This isn't a problem with launching a similar job via marathon:
{
"id": "/chris/test",
"cmd": "/usr/bin/env && /usr/bin/sleep 30",
"cpus": 0.1,
"mem": 128,
"disk": 0,
"instances": 1,
"env": {
"MY_FILEPATH": "/home/john/temp_output/test.csv"
}
}
stdout:
Registered executor on 10.0.2.56
Starting task chris_test.4fa0ac52-3eed-11e6-99b5-06094a33b9e2
sh -c '/usr/bin/env && /usr/bin/sleep 30'
Forked command at 7174
...
MY_FILEPATH=/home/john/temp_output/test.csv
...
So, I would assume it's something in mesos-exec. I haven't found the code
that does it, so I can't figure out how to subvert it.
On Thu, Jun 30, 2016 at 1:52 PM haosdent <ha...@gmail.com> wrote:
> May you show the content of /home/john/mesos/error_msg.py I would like
> to try it in my machine instead of executing `env` command.
>
> On Fri, Jul 1, 2016 at 1:40 AM, June Taylor <ju...@umn.edu> wrote:
>
>> Both of these suggestions are still resulting in this content being seen
>> by the running application: \\/home\\/john\\/temp_output\\/test.csv"
>>
>> Thank you for your continued suggestions. Please try them out yourself
>> and let me know if you can get a non-escaped path to print out successfully
>> in your executing program, and let me know what you use to do that.
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker <ch...@galacticfog.com>
>> wrote:
>>
>>> +1
>>>
>>> On Thu, Jun 30, 2016 at 1:15 PM haosdent <ha...@gmail.com> wrote:
>>>
>>>> Actually you could write your json into a file and pass the file path
>>>> into the flag. For example
>>>>
>>>> ```
>>>> $ cat /tmp/a.json
>>>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>> ```
>>>>
>>>> And use `file:///tmp/a.json` as the value of env flag.
>>>>
>>>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <ew...@groupon.com>
>>>> wrote:
>>>>
>>>>> Heh, fair enough. I know for the master/agent daemons you can put the
>>>>> config values into files, which makes this pretty clean (though
>>>>> that probably depends on the OS/init-launcher-daemon you are using). But
>>>>> yeah, when using the CLI you have to be pretty careful.
>>>>>
>>>>> - Erik
>>>>>
>>>>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com>
>>>>> wrote:
>>>>>
>>>>>> On a side note.... requiring people to put JSON on the command line
>>>>>> is a sadistic thing to do.
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <ew...@groupon.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 I would wrap every string in quotes... otherwise your shell
>>>>>>> doesn't know what you mean. i.e.,
>>>>>>>
>>>>>>> How is the shell supposed to know that you want this all to be 1
>>>>>>> string value for the --command parameter?
>>>>>>>
>>>>>>> --command=/home/john/anaconda3/bin/python
>>>>>>> /home/john/mesos/error_msg.py read
>>>>>>>
>>>>>>> Similarly how is the shell to know that you *don't* want to end this
>>>>>>> command and start a new one when you have a bare semicolon?
>>>>>>>
>>>>>>> --resources=cpus:1;mem:128
>>>>>>>
>>>>>>> - Erik
>>>>>>>
>>>>>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I use bash as well. How about
>>>>>>>> ```
>>>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>>>>> --command="/home/john/anaconda3/bin/python
>>>>>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>> ```
>>>>>>>>
>>>>>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>>>
>>>>>>>>> hasodent,
>>>>>>>>>
>>>>>>>>> We've tried your suggestion but it seems like that's not working
>>>>>>>>> out:
>>>>>>>>>
>>>>>>>>> mesos-execute --master=cluster.example.com:5050
>>>>>>>>> --name=test-program --command=/home/john/anaconda3/bin/python
>>>>>>>>> /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128
>>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>>
>>>>>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>>>>>>>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error
>>>>>>>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We're using the Bash shell. Is anyone else passing file-paths
>>>>>>>>> within their environment variable argument?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> June Taylor
>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>> University of Minnesota
>>>>>>>>>
>>>>>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <
>>>>>>>>>> chris@galacticfog.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> It's your shell:
>>>>>>>>>>>
>>>>>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> hasodent,
>>>>>>>>>>>>
>>>>>>>>>>>> I admit I don't see much of a difference in your version. Are
>>>>>>>>>>>> you saying that Mesos doesn't escape the values, and that our program may
>>>>>>>>>>>> be doing that on the executor end?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> June Taylor
>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, @Taylor I use
>>>>>>>>>>>>>
>>>>>>>>>>>>> ```
>>>>>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>>>>>>>> --name="test"
>>>>>>>>>>>>> ```
>>>>>>>>>>>>>
>>>>>>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>>>>>>
>>>>>>>>>>>>> ```
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>>>>>>
>>>>>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> ```
>>>>>>>>>>>>>
>>>>>>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Greetings,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We are using mesos-execute and passing an argument that
>>>>>>>>>>>>>> contains a Unix filepath, as follows:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>>>>>>>> executor:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>>>>>>> shutdown*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Are other people experiencing this? What is the method to
>>>>>>>>>>>>>> avoid it?
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> June Taylor
>>>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Haosdent Huang
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Haosdent Huang
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Haosdent Huang
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by haosdent <ha...@gmail.com>.
May you show the content of /home/john/mesos/error_msg.py I would like to
try it in my machine instead of executing `env` command.
On Fri, Jul 1, 2016 at 1:40 AM, June Taylor <ju...@umn.edu> wrote:
> Both of these suggestions are still resulting in this content being seen
> by the running application: \\/home\\/john\\/temp_output\\/test.csv"
>
> Thank you for your continued suggestions. Please try them out yourself and
> let me know if you can get a non-escaped path to print out successfully in
> your executing program, and let me know what you use to do that.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker <ch...@galacticfog.com>
> wrote:
>
>> +1
>>
>> On Thu, Jun 30, 2016 at 1:15 PM haosdent <ha...@gmail.com> wrote:
>>
>>> Actually you could write your json into a file and pass the file path
>>> into the flag. For example
>>>
>>> ```
>>> $ cat /tmp/a.json
>>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>> ```
>>>
>>> And use `file:///tmp/a.json` as the value of env flag.
>>>
>>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <ew...@groupon.com>
>>> wrote:
>>>
>>>> Heh, fair enough. I know for the master/agent daemons you can put the
>>>> config values into files, which makes this pretty clean (though
>>>> that probably depends on the OS/init-launcher-daemon you are using). But
>>>> yeah, when using the CLI you have to be pretty careful.
>>>>
>>>> - Erik
>>>>
>>>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com>
>>>> wrote:
>>>>
>>>>> On a side note.... requiring people to put JSON on the command line is
>>>>> a sadistic thing to do.
>>>>>
>>>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <ew...@groupon.com>
>>>>> wrote:
>>>>>
>>>>>> +1 I would wrap every string in quotes... otherwise your shell
>>>>>> doesn't know what you mean. i.e.,
>>>>>>
>>>>>> How is the shell supposed to know that you want this all to be 1
>>>>>> string value for the --command parameter?
>>>>>>
>>>>>> --command=/home/john/anaconda3/bin/python
>>>>>> /home/john/mesos/error_msg.py read
>>>>>>
>>>>>> Similarly how is the shell to know that you *don't* want to end this
>>>>>> command and start a new one when you have a bare semicolon?
>>>>>>
>>>>>> --resources=cpus:1;mem:128
>>>>>>
>>>>>> - Erik
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com> wrote:
>>>>>>
>>>>>>> I use bash as well. How about
>>>>>>> ```
>>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>>>> --command="/home/john/anaconda3/bin/python
>>>>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>> ```
>>>>>>>
>>>>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>>
>>>>>>>> hasodent,
>>>>>>>>
>>>>>>>> We've tried your suggestion but it seems like that's not working
>>>>>>>> out:
>>>>>>>>
>>>>>>>> mesos-execute --master=cluster.example.com:5050
>>>>>>>> --name=test-program --command=/home/john/anaconda3/bin/python
>>>>>>>> /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128
>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>
>>>>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>>>>>>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error
>>>>>>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>
>>>>>>>>
>>>>>>>> We're using the Bash shell. Is anyone else passing file-paths
>>>>>>>> within their environment variable argument?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> June Taylor
>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>> University of Minnesota
>>>>>>>>
>>>>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>>>>>
>>>>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <
>>>>>>>>> chris@galacticfog.com> wrote:
>>>>>>>>>
>>>>>>>>>> It's your shell:
>>>>>>>>>>
>>>>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> hasodent,
>>>>>>>>>>>
>>>>>>>>>>> I admit I don't see much of a difference in your version. Are
>>>>>>>>>>> you saying that Mesos doesn't escape the values, and that our program may
>>>>>>>>>>> be doing that on the executor end?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> June Taylor
>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, @Taylor I use
>>>>>>>>>>>>
>>>>>>>>>>>> ```
>>>>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>>>>>>> --name="test"
>>>>>>>>>>>> ```
>>>>>>>>>>>>
>>>>>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>>>>>
>>>>>>>>>>>> ```
>>>>>>>>>>>> ...
>>>>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>>>>>
>>>>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>>>>>> ...
>>>>>>>>>>>> ```
>>>>>>>>>>>>
>>>>>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Greetings,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are using mesos-execute and passing an argument that
>>>>>>>>>>>>> contains a Unix filepath, as follows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>>>>>
>>>>>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>>>>>>> executor:
>>>>>>>>>>>>>
>>>>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>>>>>> shutdown*
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are other people experiencing this? What is the method to
>>>>>>>>>>>>> avoid it?
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> June Taylor
>>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Haosdent Huang
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Haosdent Huang
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Haosdent Huang
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>
--
Best Regards,
Haosdent Huang
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by June Taylor <ju...@umn.edu>.
Both of these suggestions are still resulting in this content being seen by
the running application: \\/home\\/john\\/temp_output\\/test.csv"
Thank you for your continued suggestions. Please try them out yourself and
let me know if you can get a non-escaped path to print out successfully in
your executing program, and let me know what you use to do that.
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker <ch...@galacticfog.com> wrote:
> +1
>
> On Thu, Jun 30, 2016 at 1:15 PM haosdent <ha...@gmail.com> wrote:
>
>> Actually you could write your json into a file and pass the file path
>> into the flag. For example
>>
>> ```
>> $ cat /tmp/a.json
>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>> ```
>>
>> And use `file:///tmp/a.json` as the value of env flag.
>>
>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>>> Heh, fair enough. I know for the master/agent daemons you can put the
>>> config values into files, which makes this pretty clean (though
>>> that probably depends on the OS/init-launcher-daemon you are using). But
>>> yeah, when using the CLI you have to be pretty careful.
>>>
>>> - Erik
>>>
>>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com>
>>> wrote:
>>>
>>>> On a side note.... requiring people to put JSON on the command line is
>>>> a sadistic thing to do.
>>>>
>>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <ew...@groupon.com>
>>>> wrote:
>>>>
>>>>> +1 I would wrap every string in quotes... otherwise your shell
>>>>> doesn't know what you mean. i.e.,
>>>>>
>>>>> How is the shell supposed to know that you want this all to be 1
>>>>> string value for the --command parameter?
>>>>>
>>>>> --command=/home/john/anaconda3/bin/python
>>>>> /home/john/mesos/error_msg.py read
>>>>>
>>>>> Similarly how is the shell to know that you *don't* want to end this
>>>>> command and start a new one when you have a bare semicolon?
>>>>>
>>>>> --resources=cpus:1;mem:128
>>>>>
>>>>> - Erik
>>>>>
>>>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com> wrote:
>>>>>
>>>>>> I use bash as well. How about
>>>>>> ```
>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>>> --command="/home/john/anaconda3/bin/python
>>>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>> ```
>>>>>>
>>>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>
>>>>>>> hasodent,
>>>>>>>
>>>>>>> We've tried your suggestion but it seems like that's not working out:
>>>>>>>
>>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>>>>>> "/home/john/temp_output/test.csv"}'
>>>>>>>
>>>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>>>>>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error
>>>>>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>
>>>>>>>
>>>>>>> We're using the Bash shell. Is anyone else passing file-paths within
>>>>>>> their environment variable argument?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> June Taylor
>>>>>>> System Administrator, Minnesota Population Center
>>>>>>> University of Minnesota
>>>>>>>
>>>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>>>>
>>>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <chris@galacticfog.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> It's your shell:
>>>>>>>>>
>>>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>>>>
>>>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>>>>>>>
>>>>>>>>>> hasodent,
>>>>>>>>>>
>>>>>>>>>> I admit I don't see much of a difference in your version. Are you
>>>>>>>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>>>>>>>> doing that on the executor end?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> June Taylor
>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>> University of Minnesota
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi, @Taylor I use
>>>>>>>>>>>
>>>>>>>>>>> ```
>>>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>>>>>> --name="test"
>>>>>>>>>>> ```
>>>>>>>>>>>
>>>>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>>>>
>>>>>>>>>>> ```
>>>>>>>>>>> ...
>>>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>>>>
>>>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>>>>> ...
>>>>>>>>>>> ```
>>>>>>>>>>>
>>>>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Greetings,
>>>>>>>>>>>>
>>>>>>>>>>>> We are using mesos-execute and passing an argument that
>>>>>>>>>>>> contains a Unix filepath, as follows:
>>>>>>>>>>>>
>>>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>>>>
>>>>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>>>>>> executor:
>>>>>>>>>>>>
>>>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>>>>
>>>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>>>>> shutdown*
>>>>>>>>>>>>
>>>>>>>>>>>> Are other people experiencing this? What is the method to avoid
>>>>>>>>>>>> it?
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> June Taylor
>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Haosdent Huang
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Haosdent Huang
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Haosdent Huang
>>>>>>
>>>>>
>>>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by Chris Baker <ch...@galacticfog.com>.
+1
On Thu, Jun 30, 2016 at 1:15 PM haosdent <ha...@gmail.com> wrote:
> Actually you could write your json into a file and pass the file path into
> the flag. For example
>
> ```
> $ cat /tmp/a.json
> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
> ```
>
> And use `file:///tmp/a.json` as the value of env flag.
>
> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <ew...@groupon.com>
> wrote:
>
>> Heh, fair enough. I know for the master/agent daemons you can put the
>> config values into files, which makes this pretty clean (though
>> that probably depends on the OS/init-launcher-daemon you are using). But
>> yeah, when using the CLI you have to be pretty careful.
>>
>> - Erik
>>
>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com>
>> wrote:
>>
>>> On a side note.... requiring people to put JSON on the command line is a
>>> sadistic thing to do.
>>>
>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <ew...@groupon.com>
>>> wrote:
>>>
>>>> +1 I would wrap every string in quotes... otherwise your shell doesn't
>>>> know what you mean. i.e.,
>>>>
>>>> How is the shell supposed to know that you want this all to be 1 string
>>>> value for the --command parameter?
>>>>
>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>> read
>>>>
>>>> Similarly how is the shell to know that you *don't* want to end this
>>>> command and start a new one when you have a bare semicolon?
>>>>
>>>> --resources=cpus:1;mem:128
>>>>
>>>> - Erik
>>>>
>>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com> wrote:
>>>>
>>>>> I use bash as well. How about
>>>>> ```
>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>> --command="/home/john/anaconda3/bin/python
>>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>> ```
>>>>>
>>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>
>>>>>> hasodent,
>>>>>>
>>>>>> We've tried your suggestion but it seems like that's not working out:
>>>>>>
>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>>>>> "/home/john/temp_output/test.csv"}'
>>>>>>
>>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>>>>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error
>>>>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>
>>>>>>
>>>>>> We're using the Bash shell. Is anyone else passing file-paths within
>>>>>> their environment variable argument?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> June Taylor
>>>>>> System Administrator, Minnesota Population Center
>>>>>> University of Minnesota
>>>>>>
>>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com> wrote:
>>>>>>
>>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>>>
>>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> It's your shell:
>>>>>>>>
>>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>>>
>>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>>>>>>
>>>>>>>>> hasodent,
>>>>>>>>>
>>>>>>>>> I admit I don't see much of a difference in your version. Are you
>>>>>>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>>>>>>> doing that on the executor end?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> June Taylor
>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>> University of Minnesota
>>>>>>>>>
>>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, @Taylor I use
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>>>>> --name="test"
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> ...
>>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>>>
>>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>>>> ...
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Greetings,
>>>>>>>>>>>
>>>>>>>>>>> We are using mesos-execute and passing an argument that contains
>>>>>>>>>>> a Unix filepath, as follows:
>>>>>>>>>>>
>>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>>>
>>>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>>>>> executor:
>>>>>>>>>>>
>>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>>>
>>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>>>> shutdown*
>>>>>>>>>>>
>>>>>>>>>>> Are other people experiencing this? What is the method to avoid
>>>>>>>>>>> it?
>>>>>>>>>>> Thanks,
>>>>>>>>>>> June Taylor
>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Haosdent Huang
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Haosdent Huang
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Haosdent Huang
>>>>>
>>>>
>>>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by haosdent <ha...@gmail.com>.
Actually you could write your json into a file and pass the file path into
the flag. For example
```
$ cat /tmp/a.json
{"MY_FILEPATH": "/home/john/temp_output/test.csv"}
```
And use `file:///tmp/a.json` as the value of env flag.
On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <ew...@groupon.com>
wrote:
> Heh, fair enough. I know for the master/agent daemons you can put the
> config values into files, which makes this pretty clean (though
> that probably depends on the OS/init-launcher-daemon you are using). But
> yeah, when using the CLI you have to be pretty careful.
>
> - Erik
>
> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com>
> wrote:
>
>> On a side note.... requiring people to put JSON on the command line is a
>> sadistic thing to do.
>>
>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>>> +1 I would wrap every string in quotes... otherwise your shell doesn't
>>> know what you mean. i.e.,
>>>
>>> How is the shell supposed to know that you want this all to be 1 string
>>> value for the --command parameter?
>>>
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read
>>>
>>> Similarly how is the shell to know that you *don't* want to end this
>>> command and start a new one when you have a bare semicolon?
>>>
>>> --resources=cpus:1;mem:128
>>>
>>> - Erik
>>>
>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com> wrote:
>>>
>>>> I use bash as well. How about
>>>> ```
>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>> --command="/home/john/anaconda3/bin/python
>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>> ```
>>>>
>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>>>
>>>>> hasodent,
>>>>>
>>>>> We've tried your suggestion but it seems like that's not working out:
>>>>>
>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>>>> "/home/john/temp_output/test.csv"}'
>>>>>
>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>>>>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>>>>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>
>>>>>
>>>>> We're using the Bash shell. Is anyone else passing file-paths within
>>>>> their environment variable argument?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> June Taylor
>>>>> System Administrator, Minnesota Population Center
>>>>> University of Minnesota
>>>>>
>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com> wrote:
>>>>>
>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
>>>>>> wrote:
>>>>>>
>>>>>>> It's your shell:
>>>>>>>
>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>>
>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>>>>>
>>>>>>>> hasodent,
>>>>>>>>
>>>>>>>> I admit I don't see much of a difference in your version. Are you
>>>>>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>>>>>> doing that on the executor end?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> June Taylor
>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>> University of Minnesota
>>>>>>>>
>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi, @Taylor I use
>>>>>>>>>
>>>>>>>>> ```
>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>>>> --name="test"
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>>
>>>>>>>>> ```
>>>>>>>>> ...
>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>>
>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>>> ...
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>>
>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>>>>
>>>>>>>>>> Greetings,
>>>>>>>>>>
>>>>>>>>>> We are using mesos-execute and passing an argument that contains
>>>>>>>>>> a Unix filepath, as follows:
>>>>>>>>>>
>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>>
>>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>>>> executor:
>>>>>>>>>>
>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>>
>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>>> shutdown*
>>>>>>>>>>
>>>>>>>>>> Are other people experiencing this? What is the method to avoid
>>>>>>>>>> it?
>>>>>>>>>> Thanks,
>>>>>>>>>> June Taylor
>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>> University of Minnesota
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Haosdent Huang
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Haosdent Huang
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>
>>>
>
--
Best Regards,
Haosdent Huang
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by Erik Weathers <ew...@groupon.com>.
Heh, fair enough. I know for the master/agent daemons you can put the
config values into files, which makes this pretty clean (though
that probably depends on the OS/init-launcher-daemon you are using). But
yeah, when using the CLI you have to be pretty careful.
- Erik
On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com> wrote:
> On a side note.... requiring people to put JSON on the command line is a
> sadistic thing to do.
>
> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <ew...@groupon.com>
> wrote:
>
>> +1 I would wrap every string in quotes... otherwise your shell doesn't
>> know what you mean. i.e.,
>>
>> How is the shell supposed to know that you want this all to be 1 string
>> value for the --command parameter?
>>
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read
>>
>> Similarly how is the shell to know that you *don't* want to end this
>> command and start a new one when you have a bare semicolon?
>>
>> --resources=cpus:1;mem:128
>>
>> - Erik
>>
>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com> wrote:
>>
>>> I use bash as well. How about
>>> ```
>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>> --command="/home/john/anaconda3/bin/python
>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>> ```
>>>
>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>>
>>>> hasodent,
>>>>
>>>> We've tried your suggestion but it seems like that's not working out:
>>>>
>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>>> "/home/john/temp_output/test.csv"}'
>>>>
>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>>>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>>>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>
>>>>
>>>> We're using the Bash shell. Is anyone else passing file-paths within
>>>> their environment variable argument?
>>>>
>>>>
>>>> Thanks,
>>>> June Taylor
>>>> System Administrator, Minnesota Population Center
>>>> University of Minnesota
>>>>
>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com> wrote:
>>>>
>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>
>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
>>>>> wrote:
>>>>>
>>>>>> It's your shell:
>>>>>>
>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>
>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>>>>
>>>>>>> hasodent,
>>>>>>>
>>>>>>> I admit I don't see much of a difference in your version. Are you
>>>>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>>>>> doing that on the executor end?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> June Taylor
>>>>>>> System Administrator, Minnesota Population Center
>>>>>>> University of Minnesota
>>>>>>>
>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, @Taylor I use
>>>>>>>>
>>>>>>>> ```
>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>>> --name="test"
>>>>>>>> ```
>>>>>>>>
>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>
>>>>>>>> ```
>>>>>>>> ...
>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>
>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>> ...
>>>>>>>> ```
>>>>>>>>
>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>
>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>>>
>>>>>>>>> Greetings,
>>>>>>>>>
>>>>>>>>> We are using mesos-execute and passing an argument that contains a
>>>>>>>>> Unix filepath, as follows:
>>>>>>>>>
>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>
>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>>> executor:
>>>>>>>>>
>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>
>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>> shutdown*
>>>>>>>>>
>>>>>>>>> Are other people experiencing this? What is the method to avoid it?
>>>>>>>>> Thanks,
>>>>>>>>> June Taylor
>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>> University of Minnesota
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Haosdent Huang
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Haosdent Huang
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by Chris Baker <ch...@galacticfog.com>.
On a side note.... requiring people to put JSON on the command line is a
sadistic thing to do.
On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <ew...@groupon.com>
wrote:
> +1 I would wrap every string in quotes... otherwise your shell doesn't
> know what you mean. i.e.,
>
> How is the shell supposed to know that you want this all to be 1 string
> value for the --command parameter?
>
> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read
>
> Similarly how is the shell to know that you *don't* want to end this
> command and start a new one when you have a bare semicolon?
>
> --resources=cpus:1;mem:128
>
> - Erik
>
> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com> wrote:
>
>> I use bash as well. How about
>> ```
>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>> --command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}'
>> ```
>>
>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>>
>>> hasodent,
>>>
>>> We've tried your suggestion but it seems like that's not working out:
>>>
>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}'
>>>
>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>
>>>
>>> We're using the Bash shell. Is anyone else passing file-paths within
>>> their environment variable argument?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com> wrote:
>>>
>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>
>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
>>>> wrote:
>>>>
>>>>> It's your shell:
>>>>>
>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>
>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>>>
>>>>>> hasodent,
>>>>>>
>>>>>> I admit I don't see much of a difference in your version. Are you
>>>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>>>> doing that on the executor end?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> June Taylor
>>>>>> System Administrator, Minnesota Population Center
>>>>>> University of Minnesota
>>>>>>
>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi, @Taylor I use
>>>>>>>
>>>>>>> ```
>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>>> --name="test"
>>>>>>> ```
>>>>>>>
>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>
>>>>>>> ```
>>>>>>> ...
>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>
>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>> ...
>>>>>>> ```
>>>>>>>
>>>>>>> Not sure if your problem caused by your program
>>>>>>>
>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>>
>>>>>>>> Greetings,
>>>>>>>>
>>>>>>>> We are using mesos-execute and passing an argument that contains a
>>>>>>>> Unix filepath, as follows:
>>>>>>>>
>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>
>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>>> executor:
>>>>>>>>
>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>
>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>> shutdown*
>>>>>>>>
>>>>>>>> Are other people experiencing this? What is the method to avoid it?
>>>>>>>> Thanks,
>>>>>>>> June Taylor
>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>> University of Minnesota
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Haosdent Huang
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by Erik Weathers <ew...@groupon.com>.
+1 I would wrap every string in quotes... otherwise your shell doesn't
know what you mean. i.e.,
How is the shell supposed to know that you want this all to be 1 string
value for the --command parameter?
--command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py read
Similarly how is the shell to know that you *don't* want to end this
command and start a new one when you have a bare semicolon?
--resources=cpus:1;mem:128
- Erik
On Thu, Jun 30, 2016 at 9:08 AM, haosdent <ha...@gmail.com> wrote:
> I use bash as well. How about
> ```
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}'
> ```
>
> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
>
>> hasodent,
>>
>> We've tried your suggestion but it seems like that's not working out:
>>
>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}'
>>
>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>
>>
>> We're using the Bash shell. Is anyone else passing file-paths within
>> their environment variable argument?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com> wrote:
>>
>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>
>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
>>> wrote:
>>>
>>>> It's your shell:
>>>>
>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>
>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>>
>>>>> hasodent,
>>>>>
>>>>> I admit I don't see much of a difference in your version. Are you
>>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>>> doing that on the executor end?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> June Taylor
>>>>> System Administrator, Minnesota Population Center
>>>>> University of Minnesota
>>>>>
>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com> wrote:
>>>>>
>>>>>> Hi, @Taylor I use
>>>>>>
>>>>>> ```
>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>>> --name="test"
>>>>>> ```
>>>>>>
>>>>>> to test in my env. The output looks correct in my side
>>>>>>
>>>>>> ```
>>>>>> ...
>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>
>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>> ...
>>>>>> ```
>>>>>>
>>>>>> Not sure if your problem caused by your program
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>>
>>>>>>> Greetings,
>>>>>>>
>>>>>>> We are using mesos-execute and passing an argument that contains a
>>>>>>> Unix filepath, as follows:
>>>>>>>
>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>
>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>>> executor:
>>>>>>>
>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>
>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>> shutdown*
>>>>>>>
>>>>>>> Are other people experiencing this? What is the method to avoid it?
>>>>>>> Thanks,
>>>>>>> June Taylor
>>>>>>> System Administrator, Minnesota Population Center
>>>>>>> University of Minnesota
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Haosdent Huang
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by haosdent <ha...@gmail.com>.
I use bash as well. How about
```
mesos-execute --master=cluster.example.com:5050 --name=test-program
--command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH":
"/home/john/temp_output/test.csv"}'
```
On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <ju...@umn.edu> wrote:
> hasodent,
>
> We've tried your suggestion but it seems like that's not working out:
>
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}'
>
> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>
>
> We're using the Bash shell. Is anyone else passing file-paths within their
> environment variable argument?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com> wrote:
>
>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>
>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
>> wrote:
>>
>>> It's your shell:
>>>
>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>
>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>>
>>>> hasodent,
>>>>
>>>> I admit I don't see much of a difference in your version. Are you
>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>> doing that on the executor end?
>>>>
>>>>
>>>> Thanks,
>>>> June Taylor
>>>> System Administrator, Minnesota Population Center
>>>> University of Minnesota
>>>>
>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com> wrote:
>>>>
>>>>> Hi, @Taylor I use
>>>>>
>>>>> ```
>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>>> --name="test"
>>>>> ```
>>>>>
>>>>> to test in my env. The output looks correct in my side
>>>>>
>>>>> ```
>>>>> ...
>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>
>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>> ...
>>>>> ```
>>>>>
>>>>> Not sure if your problem caused by your program
>>>>>
>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>>>>
>>>>>> Greetings,
>>>>>>
>>>>>> We are using mesos-execute and passing an argument that contains a
>>>>>> Unix filepath, as follows:
>>>>>>
>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>
>>>>>> The slashes in the file path are then apparently escaped by
>>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>>> executor:
>>>>>>
>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>
>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>>>>>>
>>>>>> Are other people experiencing this? What is the method to avoid it?
>>>>>> Thanks,
>>>>>> June Taylor
>>>>>> System Administrator, Minnesota Population Center
>>>>>> University of Minnesota
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Haosdent Huang
>>>>>
>>>>
>>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
--
Best Regards,
Haosdent Huang
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by June Taylor <ju...@umn.edu>.
hasodent,
We've tried your suggestion but it seems like that's not working out:
mesos-execute --master=cluster.example.com:5050 --name=test-program
--command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
"/home/john/temp_output/test.csv"}'
john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
We're using the Bash shell. Is anyone else passing file-paths within their
environment variable argument?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Wed, Jun 29, 2016 at 8:55 PM, haosdent <ha...@gmail.com> wrote:
> Yes, need add quotes for --env='{"key": "value"}' flag.
>
> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
> wrote:
>
>> It's your shell:
>>
>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>
>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>>
>>> hasodent,
>>>
>>> I admit I don't see much of a difference in your version. Are you saying
>>> that Mesos doesn't escape the values, and that our program may be doing
>>> that on the executor end?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com> wrote:
>>>
>>>> Hi, @Taylor I use
>>>>
>>>> ```
>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>>> --name="test"
>>>> ```
>>>>
>>>> to test in my env. The output looks correct in my side
>>>>
>>>> ```
>>>> ...
>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>
>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>> ...
>>>> ```
>>>>
>>>> Not sure if your problem caused by your program
>>>>
>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>>>
>>>>> Greetings,
>>>>>
>>>>> We are using mesos-execute and passing an argument that contains a
>>>>> Unix filepath, as follows:
>>>>>
>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>> "/home/john/temp_output/test.csv"}*
>>>>>
>>>>> The slashes in the file path are then apparently escaped by
>>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>>> executor:
>>>>>
>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>
>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>>>>>
>>>>> Are other people experiencing this? What is the method to avoid it?
>>>>> Thanks,
>>>>> June Taylor
>>>>> System Administrator, Minnesota Population Center
>>>>> University of Minnesota
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>
>>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by haosdent <ha...@gmail.com>.
Yes, need add quotes for --env='{"key": "value"}' flag.
On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com> wrote:
> It's your shell:
>
> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
> {MY_FILEPATH: /home/john/temp_output/test.csv}
>
> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
>
>> hasodent,
>>
>> I admit I don't see much of a difference in your version. Are you saying
>> that Mesos doesn't escape the values, and that our program may be doing
>> that on the executor end?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com> wrote:
>>
>>> Hi, @Taylor I use
>>>
>>> ```
>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>>> --name="test"
>>> ```
>>>
>>> to test in my env. The output looks correct in my side
>>>
>>> ```
>>> ...
>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>
>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>> ...
>>> ```
>>>
>>> Not sure if your problem caused by your program
>>>
>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>>
>>>> Greetings,
>>>>
>>>> We are using mesos-execute and passing an argument that contains a Unix
>>>> filepath, as follows:
>>>>
>>>> *mesos-execute --master=cluster.example.com:5050
>>>> <http://cluster.example.com:5050> --name=test-program
>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>> "/home/john/temp_output/test.csv"}*
>>>>
>>>> The slashes in the file path are then apparently escaped by
>>>> mesos-execute and therefore fail to be useful once picked up by the
>>>> executor:
>>>>
>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>
>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>>>>
>>>> Are other people experiencing this? What is the method to avoid it?
>>>> Thanks,
>>>> June Taylor
>>>> System Administrator, Minnesota Population Center
>>>> University of Minnesota
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>
--
Best Regards,
Haosdent Huang
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by Chris Baker <ch...@galacticfog.com>.
It's your shell:
$ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
{MY_FILEPATH: /home/john/temp_output/test.csv}
On Wed, Jun 29, 2016 at 4:05 PM June Taylor <ju...@umn.edu> wrote:
> hasodent,
>
> I admit I don't see much of a difference in your version. Are you saying
> that Mesos doesn't escape the values, and that our program may be doing
> that on the executor end?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com> wrote:
>
>> Hi, @Taylor I use
>>
>> ```
>> ./src/mesos-execute --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
>> --name="test"
>> ```
>>
>> to test in my env. The output looks correct in my side
>>
>> ```
>> ...
>> MY_FILEPATH=/home/john/temp_output/test.csv
>>
>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>> ...
>> ```
>>
>> Not sure if your problem caused by your program
>>
>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>>
>>> Greetings,
>>>
>>> We are using mesos-execute and passing an argument that contains a Unix
>>> filepath, as follows:
>>>
>>> *mesos-execute --master=cluster.example.com:5050
>>> <http://cluster.example.com:5050> --name=test-program
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}*
>>>
>>> The slashes in the file path are then apparently escaped by
>>> mesos-execute and therefore fail to be useful once picked up by the
>>> executor:
>>>
>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>
>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>>>
>>> Are other people experiencing this? What is the method to avoid it?
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by June Taylor <ju...@umn.edu>.
hasodent,
I admit I don't see much of a difference in your version. Are you saying
that Mesos doesn't escape the values, and that our program may be doing
that on the executor end?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Wed, Jun 29, 2016 at 2:02 PM, haosdent <ha...@gmail.com> wrote:
> Hi, @Taylor I use
>
> ```
> ./src/mesos-execute --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
> --name="test"
> ```
>
> to test in my env. The output looks correct in my side
>
> ```
> ...
> MY_FILEPATH=/home/john/temp_output/test.csv
>
> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
> ...
> ```
>
> Not sure if your problem caused by your program
>
> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
>
>> Greetings,
>>
>> We are using mesos-execute and passing an argument that contains a Unix
>> filepath, as follows:
>>
>> *mesos-execute --master=cluster.example.com:5050
>> <http://cluster.example.com:5050> --name=test-program
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}*
>>
>> The slashes in the file path are then apparently escaped by mesos-execute
>> and therefore fail to be useful once picked up by the executor:
>>
>> *FileNotFoundError: [Errno 2] No such file or directory:
>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>
>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>>
>> Are other people experiencing this? What is the method to avoid it?
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Escaped characters in the 'env' argument passed to mesos-execute
Posted by haosdent <ha...@gmail.com>.
Hi, @Taylor I use
```
./src/mesos-execute --env='{"MY_FILEPATH":
"/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
--name="test"
```
to test in my env. The output looks correct in my side
```
...
MY_FILEPATH=/home/john/temp_output/test.csv
MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
...
```
Not sure if your problem caused by your program
On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <ju...@umn.edu> wrote:
> Greetings,
>
> We are using mesos-execute and passing an argument that contains a Unix
> filepath, as follows:
>
> *mesos-execute --master=cluster.example.com:5050
> <http://cluster.example.com:5050> --name=test-program
> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}*
>
> The slashes in the file path are then apparently escaped by mesos-execute
> and therefore fail to be useful once picked up by the executor:
>
> *FileNotFoundError: [Errno 2] No such file or directory:
> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>
> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>
> Are other people experiencing this? What is the method to avoid it?
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
--
Best Regards,
Haosdent Huang