You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@aurora.apache.org by Krish <kr...@gmail.com> on 2016/03/16 13:54:55 UTC
Aurora Thrift API Info
Hi,
I was going through the Aurora Thrift API to determine how to add new jobs.
I am using aurora v0.12 released last month and have upgraded to mesos
v0.25 accordingly.
Below is a summary of my (very limited) understanding of some APIs, & would
help it if someone can point out flaws in my understanding:
1. All APIs require thrift inputs of the structs specified, and return
thrift values only in Response.result field.
2. Is there a set of examples in the documentation to help understand
Thrift API better?
3. createJob(JobDescription desc, Lock lock):
This is basically the API to replace the Aurora DSL/.aurora files for
job configuration.
4. What is the Lock object? I see that some APIs require locking and
some don't. For example, createJob needs a Lock object as parameter, & I am
assuming that it is required so that one does not create multiple jobs with
the same JobKey.
5. addInstances(AddInstancesConfig cfg, Lock lock):
By the naming convention, it seems this is used to increase the number
of instances of a job. It will not result in stopping of current instances
of the job.
My second explanation for this API: Since it uses a set of instanceIds,
this is used for adding already running job in slaves to the internal data
structures of Aurora to track the job.
6. getPendingResult(TaskQuery taskquery):
Return the reason (in string) about why the job is PENDING. For example:
insufficient CPU.
7. setQuota & getQuota for setting user level quotas.
8. killTasks to kill all running instances of a job in the cluster.
9. startJobUpdate(JobUpdateRequest request, string message):
Used for updating jobs with the new TaskConfig specified. Can be used if
resource requirement changes. For example: If I wanted aurora to update the
version of container used for a job using TaskConfig.Container attribute.
An aurora scheduling question is if I start a job with 5 instances, and
there are resources available to run only 4 of them, does the entire job
block, or only the 5th instance of the job blocks?
Thanks!
--
κρισhναν
Re: Aurora Thrift API Info
Posted by Maxim Khutornenko <ma...@apache.org>.
>
> Question is when I am modifying job details, in particular, scaling up
> instances based on demand, do I use the startJobUpdate or the addInstances
> API?
If all you need is scale out (meaning add more instances of the existing
config) the 'addInstances' should be an easier and faster API to work with.
The 'startJobUpdate' can still be used to add/remove instances but it's
main goal is to perform a rolling update of existing instances. So, the
guideline may be:
- addInstances: Adds instances of the existing config. Use for a quick
scale out of *existing* job, e.g. in an autoscaler solution.
- killTasks: Kills existing instances. Use to reduce the number of running
instances or kill the job entirely
- startJobUpdate: Performs rolling update of the existing job. This is a
more involved API that requires full job configuration. It's capable of
updating existing services in a safe manner by performing health evaluation
during the rollout. It's also capable of rolling back the failed update
automatically. You can also add instances of the *new *config during the
update as well as remove instances of the *old *config if needed.
Only killTasks and startJobUpdate are potentially destructive (capable of
killing instances). The addInstances never kills anything.
And if I reduce the instances (for eg, from 6 to 5), will the API
> (addInstances or startJobUpdate) also kill the last instance of the job?
The startJobUpdate ensures the job instance count matches the one specified
in the .aurora config. So, if you reduced the count from 6 to 5 (via
killTasks) AND your .aurora config still has 'instances=6' you job will be
restored back to full count. Think of startJobUpdate as the ultimate source
of truth. No matter what the current job state is (wrt instance counts or
task configs) it will attempt to bring everything to the common denominator
in terms of task_config and instance_count.
On Wed, Mar 16, 2016 at 11:36 AM, Krish <kr...@gmail.com> wrote:
> Thanks, Maxim & Bill!
>
> I would love some more clarifications to the below observations.
>
> A little googling helped me find
> https://issues.apache.org/jira/browse/AURORA-1258, which then led me to
> http://markmail.org/message/al26gmpwlcns3oer#query:+page:1+mid:2smaej5n5e54li3g+state:results
> .
>
> Question is when I am modifying job details, in particular, scaling up
> instances based on demand, do I use the startJobUpdate or the addInstances
> API?
> Seems like addInstances is supposed to do this, but you mention that
> startJobUpdate is also supposed to be the "main API to change your
> service job in any way (including adding, removing or modifying instances).
> "
>
> Also, if both are valid, under what scenarios would one use startJobUpdate?
> Which one will be non-destructive? As in, which API does not kill current
> instances while adding new ones?
>
> And if I reduce the instances (for eg, from 6 to 5), will the API
> (addInstances or startJobUpdate) also kill the last instance of the job?
>
>
> --
> κρισhναν
>
> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org> wrote:
>
>> Regarding documentation - Maxim is correct that there isn't much in the
>> way of independent/holistic docs for the thrift API. There is, however,
>> scant javadoc-style documentation within the IDL spec itself:
>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>
>> If you are looking to use the thrift API directly, the most difficult API
>> method will be defining the ExecutorConfig.data value when calling
>> createJob. Please don't hesitate to ask for assistance if you get to that
>> point!
>>
>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>> wrote:
>>
>>> 1. All APIs require thrift inputs of the structs specified, and return
>>>> thrift values only in Response.result field.
>>>
>>> Correct. There is also 'details' field that may have additional messages
>>> (of error or informational nature)
>>>
>>> 2. Is there a set of examples in the documentation to help understand
>>>> Thrift API better?
>>>
>>> The thrift API is largely undocumented. There is an effort to bring up a
>>> fully supported REST API that will presumably get documented and become
>>> much easier to use. It's mostly in flux now.
>>>
>>> 3. createJob(JobDescription desc, Lock lock):
>>>
>>> This is the API to use when you a brand new service or adhoc (batch) job
>>> created. The JobDescription is populated from the .aurora config. You may
>>> want to trace "aurora job create" client command implementation to see how
>>> it happens.
>>>
>>> 4. What is the Lock object? I see that some APIs require locking and
>>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>>> assuming that it is required so that one does not create multiple jobs with
>>>> the same JobKey.
>>>
>>> Ignore this object as it's an echo of the old client updater. It's now
>>> deprecated and will be removed soon. You can pass NULL for now.
>>>
>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>
>>> Another echo of the client updater but this time it's got a second life.
>>> Check out its new signature and comments in the api.thrift. It's
>>> essentially a "scale-out" API that can add instances to the existing job
>>> without changing the underlying task assumptions.
>>>
>>> 6. getPendingResult(TaskQuery taskquery):
>>>
>>> It's actually 'getPendingReason' and is currently used exclusively by
>>> the UI to get the reason for a task PENDING state.
>>>
>>> 7. setQuota & getQuota for setting user level quotas.
>>>
>>> This is to set role-level quota. Currently only required for tasks with
>>> 'production=True'. Search through our docs for more details.
>>>
>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>
>>> It's quite versatile and can be used to kill some or all instances of
>>> the job.
>>>
>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>
>>> Your observations are correct. This is the main API to change your
>>> service job in any way (including adding, removing or modifying instances).
>>>
>>> An aurora scheduling question is if I start a job with 5 instances, and
>>>> there are resources available to run only 4 of them, does the entire job
>>>> block, or only the 5th instance of the job blocks?
>>>
>>> Scheduler will try to schedule as many instances as it can. Those that
>>> will not find resources will remain in PENDING state until more resources
>>> are available. In your particular example only the 5th will remain PENDING.
>>>
>>>
>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I was going through the Aurora Thrift API to determine how to add new
>>>> jobs.
>>>> I am using aurora v0.12 released last month and have upgraded to mesos
>>>> v0.25 accordingly.
>>>>
>>>> Below is a summary of my (very limited) understanding of some APIs, &
>>>> would help it if someone can point out flaws in my understanding:
>>>>
>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>> return thrift values only in Response.result field.
>>>>
>>>> 2. Is there a set of examples in the documentation to help
>>>> understand Thrift API better?
>>>>
>>>> 3. createJob(JobDescription desc, Lock lock):
>>>> This is basically the API to replace the Aurora DSL/.aurora files
>>>> for job configuration.
>>>>
>>>> 4. What is the Lock object? I see that some APIs require locking
>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>> I am assuming that it is required so that one does not create multiple jobs
>>>> with the same JobKey.
>>>>
>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>> By the naming convention, it seems this is used to increase the
>>>> number of instances of a job. It will not result in stopping of current
>>>> instances of the job.
>>>>
>>>> My second explanation for this API: Since it uses a set of
>>>> instanceIds, this is used for adding already running job in slaves to the
>>>> internal data structures of Aurora to track the job.
>>>>
>>>> 6. getPendingResult(TaskQuery taskquery):
>>>> Return the reason (in string) about why the job is PENDING. For
>>>> example: insufficient CPU.
>>>>
>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>
>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>
>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>> Used for updating jobs with the new TaskConfig specified. Can be
>>>> used if resource requirement changes. For example: If I wanted aurora to
>>>> update the version of container used for a job using TaskConfig.Container
>>>> attribute.
>>>>
>>>>
>>>> An aurora scheduling question is if I start a job with 5 instances, and
>>>> there are resources available to run only 4 of them, does the entire job
>>>> block, or only the 5th instance of the job blocks?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>
>>>
>>
>
RE: Aurora Thrift API Info
Posted by "Rice, Ben" <Be...@netapp.com>.
Thank you for the reply—I’ll dig through and see what I can figure out. I was hoping to use the API to create a new job and simply pass in the .aurora file path or content, but it looks like it’ll be a little more work than that. ☺
-Ben
From: Bill Farner [mailto:wfarner@apache.org]
Sent: Wednesday, March 16, 2016 9:31 PM
To: user@aurora.apache.org
Cc: krishnan.k.iyer@gmail.com; maxim@apache.org
Subject: Re: Aurora Thrift API Info
I like the DSL that Aurora has and I was wondering if there was any way to use the API and somehow get the DSL translated in to what the ExecutorConfig structure requires?
For an ad-hoc peek at what's in the ExecutorConfig, you can follow the bit of hacking described in this thread:
http://mail-archives.apache.org/mod_mbox/aurora-dev/201601.mbox/%3CCAC2vyCXEOt1xaPC3DK_o_-J%3Dunjg7csQAajb5Ekad-g%2B20VxHA%40mail.gmail.com%3E
As for doing it via the API, you will find the executor-specific DSL contents in the response to getTasksStatus
https://github.com/apache/aurora/blob/ec0f38a7528f8adb8f0769112a1043b98598c03f/api/src/main/thrift/org/apache/aurora/gen/api.thrift#L945-L946
that API call will populate Result.scheduleStatusResult:
https://github.com/apache/aurora/blob/ec0f38a7528f8adb8f0769112a1043b98598c03f/api/src/main/thrift/org/apache/aurora/gen/api.thrift#L899
From there the chain will be ScheduledTask->AssignedTask->TaskConfig->ExecutorConfig
Not an obvious path, but it can be done!
On Wed, Mar 16, 2016 at 11:54 AM, Rice, Ben <Be...@netapp.com>> wrote:
Sorry to piggy back on the thread, but I had a related question.
I’m wanting to write a GUI for creating/managing/etc jobs. I like the DSL that Aurora has and I was wondering if there was any way to use the API and somehow get the DSL translated in to what the ExecutorConfig structure requires?
Thanks,
-Ben
From: Krish [mailto:krishnan.k.iyer@gmail.com<ma...@gmail.com>]
Sent: Wednesday, March 16, 2016 2:36 PM
To: user@aurora.apache.org<ma...@aurora.apache.org>; Bill Farner <wf...@apache.org>>; maxim@apache.org<ma...@apache.org>
Subject: Re: Aurora Thrift API Info
Thanks, Maxim & Bill!
I would love some more clarifications to the below observations.
A little googling helped me find https://issues.apache.org/jira/browse/AURORA-1258, which then led me to http://markmail.org/message/al26gmpwlcns3oer#query:+page:1+mid:2smaej5n5e54li3g+state:results.
Question is when I am modifying job details, in particular, scaling up instances based on demand, do I use the startJobUpdate or the addInstances API?
Seems like addInstances is supposed to do this, but you mention that startJobUpdate is also supposed to be the "main API to change your service job in any way (including adding, removing or modifying instances). "
Also, if both are valid, under what scenarios would one use startJobUpdate?
Which one will be non-destructive? As in, which API does not kill current instances while adding new ones?
And if I reduce the instances (for eg, from 6 to 5), will the API (addInstances or startJobUpdate) also kill the last instance of the job?
--
κρισhναν
On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>> wrote:
Regarding documentation - Maxim is correct that there isn't much in the way of independent/holistic docs for the thrift API. There is, however, scant javadoc-style documentation within the IDL spec itself: https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
If you are looking to use the thrift API directly, the most difficult API method will be defining the ExecutorConfig.data value when calling createJob. Please don't hesitate to ask for assistance if you get to that point!
On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>> wrote:
1. All APIs require thrift inputs of the structs specified, and return thrift values only in Response.result field.
Correct. There is also 'details' field that may have additional messages (of error or informational nature)
2. Is there a set of examples in the documentation to help understand Thrift API better?
The thrift API is largely undocumented. There is an effort to bring up a fully supported REST API that will presumably get documented and become much easier to use. It's mostly in flux now.
3. createJob(JobDescription desc, Lock lock):
This is the API to use when you a brand new service or adhoc (batch) job created. The JobDescription is populated from the .aurora config. You may want to trace "aurora job create" client command implementation to see how it happens.
4. What is the Lock object? I see that some APIs require locking and some don't. For example, createJob needs a Lock object as parameter, & I am assuming that it is required so that one does not create multiple jobs with the same JobKey.
Ignore this object as it's an echo of the old client updater. It's now deprecated and will be removed soon. You can pass NULL for now.
5. addInstances(AddInstancesConfig cfg, Lock lock):
Another echo of the client updater but this time it's got a second life. Check out its new signature and comments in the api.thrift. It's essentially a "scale-out" API that can add instances to the existing job without changing the underlying task assumptions.
6. getPendingResult(TaskQuery taskquery):
It's actually 'getPendingReason' and is currently used exclusively by the UI to get the reason for a task PENDING state.
7. setQuota & getQuota for setting user level quotas.
This is to set role-level quota. Currently only required for tasks with 'production=True'. Search through our docs for more details.
8. killTasks to kill all running instances of a job in the cluster.
It's quite versatile and can be used to kill some or all instances of the job.
9. startJobUpdate(JobUpdateRequest request, string message):
Your observations are correct. This is the main API to change your service job in any way (including adding, removing or modifying instances).
An aurora scheduling question is if I start a job with 5 instances, and there are resources available to run only 4 of them, does the entire job block, or only the 5th instance of the job blocks?
Scheduler will try to schedule as many instances as it can. Those that will not find resources will remain in PENDING state until more resources are available. In your particular example only the 5th will remain PENDING.
On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>> wrote:
Hi,
I was going through the Aurora Thrift API to determine how to add new jobs.
I am using aurora v0.12 released last month and have upgraded to mesos v0.25 accordingly.
Below is a summary of my (very limited) understanding of some APIs, & would help it if someone can point out flaws in my understanding:
1. All APIs require thrift inputs of the structs specified, and return thrift values only in Response.result field.
2. Is there a set of examples in the documentation to help understand Thrift API better?
3. createJob(JobDescription desc, Lock lock):
This is basically the API to replace the Aurora DSL/.aurora files for job configuration.
4. What is the Lock object? I see that some APIs require locking and some don't. For example, createJob needs a Lock object as parameter, & I am assuming that it is required so that one does not create multiple jobs with the same JobKey.
5. addInstances(AddInstancesConfig cfg, Lock lock):
By the naming convention, it seems this is used to increase the number of instances of a job. It will not result in stopping of current instances of the job.
My second explanation for this API: Since it uses a set of instanceIds, this is used for adding already running job in slaves to the internal data structures of Aurora to track the job.
6. getPendingResult(TaskQuery taskquery):
Return the reason (in string) about why the job is PENDING. For example: insufficient CPU.
7. setQuota & getQuota for setting user level quotas.
8. killTasks to kill all running instances of a job in the cluster.
9. startJobUpdate(JobUpdateRequest request, string message):
Used for updating jobs with the new TaskConfig specified. Can be used if resource requirement changes. For example: If I wanted aurora to update the version of container used for a job using TaskConfig.Container attribute.
An aurora scheduling question is if I start a job with 5 instances, and there are resources available to run only 4 of them, does the entire job block, or only the 5th instance of the job blocks?
Thanks!
--
κρισhναν
Re: Aurora Thrift API Info
Posted by Bill Farner <wf...@apache.org>.
>
> I like the DSL that Aurora has and I was wondering if there was any way to
> use the API and somehow get the DSL translated in to what the
> ExecutorConfig structure requires?
For an ad-hoc peek at what's in the ExecutorConfig, you can follow the bit
of hacking described in this thread:
http://mail-archives.apache.org/mod_mbox/aurora-dev/201601.mbox/%3CCAC2vyCXEOt1xaPC3DK_o_-J%3Dunjg7csQAajb5Ekad-g%2B20VxHA%40mail.gmail.com%3E
As for doing it via the API, you will find the executor-specific DSL
contents in the response to getTasksStatus
https://github.com/apache/aurora/blob/ec0f38a7528f8adb8f0769112a1043b98598c03f/api/src/main/thrift/org/apache/aurora/gen/api.thrift#L945-L946
that API call will populate Result.scheduleStatusResult:
https://github.com/apache/aurora/blob/ec0f38a7528f8adb8f0769112a1043b98598c03f/api/src/main/thrift/org/apache/aurora/gen/api.thrift#L899
>From there the chain will be
ScheduledTask->AssignedTask->TaskConfig->ExecutorConfig
Not an obvious path, but it can be done!
On Wed, Mar 16, 2016 at 11:54 AM, Rice, Ben <Be...@netapp.com> wrote:
> Sorry to piggy back on the thread, but I had a related question.
>
>
>
> I’m wanting to write a GUI for creating/managing/etc jobs. I like the DSL
> that Aurora has and I was wondering if there was any way to use the API and
> somehow get the DSL translated in to what the ExecutorConfig structure
> requires?
>
>
>
> Thanks,
>
> -Ben
>
>
>
> *From:* Krish [mailto:krishnan.k.iyer@gmail.com]
> *Sent:* Wednesday, March 16, 2016 2:36 PM
> *To:* user@aurora.apache.org; Bill Farner <wf...@apache.org>;
> maxim@apache.org
> *Subject:* Re: Aurora Thrift API Info
>
>
>
> Thanks, Maxim & Bill!
>
>
>
> I would love some more clarifications to the below observations.
>
>
>
> A little googling helped me find
> https://issues.apache.org/jira/browse/AURORA-1258, which then led me to
> http://markmail.org/message/al26gmpwlcns3oer#query:+page:1+mid:2smaej5n5e54li3g+state:results
> .
>
>
>
> Question is when I am modifying job details, in particular, scaling up
> instances based on demand, do I use the startJobUpdate or the addInstances
> API?
>
> Seems like addInstances is supposed to do this, but you mention that
> startJobUpdate is also supposed to be the "main API to change your
> service job in any way (including adding, removing or modifying instances).
> "
>
>
>
> Also, if both are valid, under what scenarios would one use startJobUpdate?
>
> Which one will be non-destructive? As in, which API does not kill current
> instances while adding new ones?
>
>
>
> And if I reduce the instances (for eg, from 6 to 5), will the API
> (addInstances or startJobUpdate) also kill the last instance of the job?
>
>
>
>
>
> --
>
> κρισhναν
>
>
>
> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org> wrote:
>
> Regarding documentation - Maxim is correct that there isn't much in the
> way of independent/holistic docs for the thrift API. There is, however,
> scant javadoc-style documentation within the IDL spec itself:
> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>
>
>
> If you are looking to use the thrift API directly, the most difficult API
> method will be defining the ExecutorConfig.data value when calling
> createJob. Please don't hesitate to ask for assistance if you get to that
> point!
>
>
>
> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
>
> 1. All APIs require thrift inputs of the structs specified, and return
> thrift values only in Response.result field.
>
> Correct. There is also 'details' field that may have additional messages
> (of error or informational nature)
>
>
>
> 2. Is there a set of examples in the documentation to help understand
> Thrift API better?
>
> The thrift API is largely undocumented. There is an effort to bring up a
> fully supported REST API that will presumably get documented and become
> much easier to use. It's mostly in flux now.
>
>
>
> 3. createJob(JobDescription desc, Lock lock):
>
> This is the API to use when you a brand new service or adhoc (batch) job
> created. The JobDescription is populated from the .aurora config. You may
> want to trace "aurora job create" client command implementation to see how
> it happens.
>
>
>
> 4. What is the Lock object? I see that some APIs require locking and some
> don't. For example, createJob needs a Lock object as parameter, & I am
> assuming that it is required so that one does not create multiple jobs with
> the same JobKey.
>
> Ignore this object as it's an echo of the old client updater. It's now
> deprecated and will be removed soon. You can pass NULL for now.
>
>
>
> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>
> Another echo of the client updater but this time it's got a second life.
> Check out its new signature and comments in the api.thrift. It's
> essentially a "scale-out" API that can add instances to the existing job
> without changing the underlying task assumptions.
>
>
>
> 6. getPendingResult(TaskQuery taskquery):
>
> It's actually 'getPendingReason' and is currently used exclusively by the
> UI to get the reason for a task PENDING state.
>
>
>
> 7. setQuota & getQuota for setting user level quotas.
>
> This is to set role-level quota. Currently only required for tasks with
> 'production=True'. Search through our docs for more details.
>
>
>
> 8. killTasks to kill all running instances of a job in the cluster.
>
> It's quite versatile and can be used to kill some or all instances of the
> job.
>
>
>
> 9. startJobUpdate(JobUpdateRequest request, string message):
>
> Your observations are correct. This is the main API to change your service
> job in any way (including adding, removing or modifying instances).
>
>
>
> An aurora scheduling question is if I start a job with 5 instances, and
> there are resources available to run only 4 of them, does the entire job
> block, or only the 5th instance of the job blocks?
>
> Scheduler will try to schedule as many instances as it can. Those that
> will not find resources will remain in PENDING state until more resources
> are available. In your particular example only the 5th will remain PENDING.
>
>
>
>
>
> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com> wrote:
>
> Hi,
>
> I was going through the Aurora Thrift API to determine how to add new jobs.
>
> I am using aurora v0.12 released last month and have upgraded to mesos
> v0.25 accordingly.
>
>
>
> Below is a summary of my (very limited) understanding of some APIs, &
> would help it if someone can point out flaws in my understanding:
>
> 1. All APIs require thrift inputs of the structs specified, and return
> thrift values only in Response.result field.
> 2. Is there a set of examples in the documentation to help understand
> Thrift API better?
> 3. createJob(JobDescription desc, Lock lock):
> This is basically the API to replace the Aurora DSL/.aurora files for
> job configuration.
> 4. What is the Lock object? I see that some APIs require locking and
> some don't. For example, createJob needs a Lock object as parameter, & I am
> assuming that it is required so that one does not create multiple jobs with
> the same JobKey.
> 5. addInstances(AddInstancesConfig cfg, Lock lock):
> By the naming convention, it seems this is used to increase the number
> of instances of a job. It will not result in stopping of current instances
> of the job.
>
> My second explanation for this API: Since it uses a set of
> instanceIds, this is used for adding already running job in slaves to the
> internal data structures of Aurora to track the job.
> 6. getPendingResult(TaskQuery taskquery):
> Return the reason (in string) about why the job is PENDING. For
> example: insufficient CPU.
> 7. setQuota & getQuota for setting user level quotas.
> 8. killTasks to kill all running instances of a job in the cluster.
> 9. startJobUpdate(JobUpdateRequest request, string message):
> Used for updating jobs with the new TaskConfig specified. Can be used
> if resource requirement changes. For example: If I wanted aurora to update
> the version of container used for a job using TaskConfig.Container
> attribute.
>
> An aurora scheduling question is if I start a job with 5 instances, and
> there are resources available to run only 4 of them, does the entire job
> block, or only the 5th instance of the job blocks?
>
>
>
> Thanks!
>
>
>
> --
>
> κρισhναν
>
>
>
>
>
>
>
RE: Aurora Thrift API Info
Posted by "Rice, Ben" <Be...@netapp.com>.
Sorry to piggy back on the thread, but I had a related question.
I’m wanting to write a GUI for creating/managing/etc jobs. I like the DSL that Aurora has and I was wondering if there was any way to use the API and somehow get the DSL translated in to what the ExecutorConfig structure requires?
Thanks,
-Ben
From: Krish [mailto:krishnan.k.iyer@gmail.com]
Sent: Wednesday, March 16, 2016 2:36 PM
To: user@aurora.apache.org; Bill Farner <wf...@apache.org>; maxim@apache.org
Subject: Re: Aurora Thrift API Info
Thanks, Maxim & Bill!
I would love some more clarifications to the below observations.
A little googling helped me find https://issues.apache.org/jira/browse/AURORA-1258, which then led me to http://markmail.org/message/al26gmpwlcns3oer#query:+page:1+mid:2smaej5n5e54li3g+state:results.
Question is when I am modifying job details, in particular, scaling up instances based on demand, do I use the startJobUpdate or the addInstances API?
Seems like addInstances is supposed to do this, but you mention that startJobUpdate is also supposed to be the "main API to change your service job in any way (including adding, removing or modifying instances). "
Also, if both are valid, under what scenarios would one use startJobUpdate?
Which one will be non-destructive? As in, which API does not kill current instances while adding new ones?
And if I reduce the instances (for eg, from 6 to 5), will the API (addInstances or startJobUpdate) also kill the last instance of the job?
--
κρισhναν
On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>> wrote:
Regarding documentation - Maxim is correct that there isn't much in the way of independent/holistic docs for the thrift API. There is, however, scant javadoc-style documentation within the IDL spec itself: https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
If you are looking to use the thrift API directly, the most difficult API method will be defining the ExecutorConfig.data value when calling createJob. Please don't hesitate to ask for assistance if you get to that point!
On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>> wrote:
1. All APIs require thrift inputs of the structs specified, and return thrift values only in Response.result field.
Correct. There is also 'details' field that may have additional messages (of error or informational nature)
2. Is there a set of examples in the documentation to help understand Thrift API better?
The thrift API is largely undocumented. There is an effort to bring up a fully supported REST API that will presumably get documented and become much easier to use. It's mostly in flux now.
3. createJob(JobDescription desc, Lock lock):
This is the API to use when you a brand new service or adhoc (batch) job created. The JobDescription is populated from the .aurora config. You may want to trace "aurora job create" client command implementation to see how it happens.
4. What is the Lock object? I see that some APIs require locking and some don't. For example, createJob needs a Lock object as parameter, & I am assuming that it is required so that one does not create multiple jobs with the same JobKey.
Ignore this object as it's an echo of the old client updater. It's now deprecated and will be removed soon. You can pass NULL for now.
5. addInstances(AddInstancesConfig cfg, Lock lock):
Another echo of the client updater but this time it's got a second life. Check out its new signature and comments in the api.thrift. It's essentially a "scale-out" API that can add instances to the existing job without changing the underlying task assumptions.
6. getPendingResult(TaskQuery taskquery):
It's actually 'getPendingReason' and is currently used exclusively by the UI to get the reason for a task PENDING state.
7. setQuota & getQuota for setting user level quotas.
This is to set role-level quota. Currently only required for tasks with 'production=True'. Search through our docs for more details.
8. killTasks to kill all running instances of a job in the cluster.
It's quite versatile and can be used to kill some or all instances of the job.
9. startJobUpdate(JobUpdateRequest request, string message):
Your observations are correct. This is the main API to change your service job in any way (including adding, removing or modifying instances).
An aurora scheduling question is if I start a job with 5 instances, and there are resources available to run only 4 of them, does the entire job block, or only the 5th instance of the job blocks?
Scheduler will try to schedule as many instances as it can. Those that will not find resources will remain in PENDING state until more resources are available. In your particular example only the 5th will remain PENDING.
On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>> wrote:
Hi,
I was going through the Aurora Thrift API to determine how to add new jobs.
I am using aurora v0.12 released last month and have upgraded to mesos v0.25 accordingly.
Below is a summary of my (very limited) understanding of some APIs, & would help it if someone can point out flaws in my understanding:
1. All APIs require thrift inputs of the structs specified, and return thrift values only in Response.result field.
2. Is there a set of examples in the documentation to help understand Thrift API better?
3. createJob(JobDescription desc, Lock lock):
This is basically the API to replace the Aurora DSL/.aurora files for job configuration.
4. What is the Lock object? I see that some APIs require locking and some don't. For example, createJob needs a Lock object as parameter, & I am assuming that it is required so that one does not create multiple jobs with the same JobKey.
5. addInstances(AddInstancesConfig cfg, Lock lock):
By the naming convention, it seems this is used to increase the number of instances of a job. It will not result in stopping of current instances of the job.
My second explanation for this API: Since it uses a set of instanceIds, this is used for adding already running job in slaves to the internal data structures of Aurora to track the job.
6. getPendingResult(TaskQuery taskquery):
Return the reason (in string) about why the job is PENDING. For example: insufficient CPU.
7. setQuota & getQuota for setting user level quotas.
8. killTasks to kill all running instances of a job in the cluster.
9. startJobUpdate(JobUpdateRequest request, string message):
Used for updating jobs with the new TaskConfig specified. Can be used if resource requirement changes. For example: If I wanted aurora to update the version of container used for a job using TaskConfig.Container attribute.
An aurora scheduling question is if I start a job with 5 instances, and there are resources available to run only 4 of them, does the entire job block, or only the 5th instance of the job blocks?
Thanks!
--
κρισhναν
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Thanks, Maxim & Bill!
I would love some more clarifications to the below observations.
A little googling helped me find
https://issues.apache.org/jira/browse/AURORA-1258, which then led me to
http://markmail.org/message/al26gmpwlcns3oer#query:+page:1+mid:2smaej5n5e54li3g+state:results
.
Question is when I am modifying job details, in particular, scaling up
instances based on demand, do I use the startJobUpdate or the addInstances
API?
Seems like addInstances is supposed to do this, but you mention that
startJobUpdate is also supposed to be the "main API to change your service
job in any way (including adding, removing or modifying instances). "
Also, if both are valid, under what scenarios would one use startJobUpdate?
Which one will be non-destructive? As in, which API does not kill current
instances while adding new ones?
And if I reduce the instances (for eg, from 6 to 5), will the API
(addInstances or startJobUpdate) also kill the last instance of the job?
--
κρισhναν
On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org> wrote:
> Regarding documentation - Maxim is correct that there isn't much in the
> way of independent/holistic docs for the thrift API. There is, however,
> scant javadoc-style documentation within the IDL spec itself:
> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>
> If you are looking to use the thrift API directly, the most difficult API
> method will be defining the ExecutorConfig.data value when calling
> createJob. Please don't hesitate to ask for assistance if you get to that
> point!
>
> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
>
>> 1. All APIs require thrift inputs of the structs specified, and return
>>> thrift values only in Response.result field.
>>
>> Correct. There is also 'details' field that may have additional messages
>> (of error or informational nature)
>>
>> 2. Is there a set of examples in the documentation to help understand
>>> Thrift API better?
>>
>> The thrift API is largely undocumented. There is an effort to bring up a
>> fully supported REST API that will presumably get documented and become
>> much easier to use. It's mostly in flux now.
>>
>> 3. createJob(JobDescription desc, Lock lock):
>>
>> This is the API to use when you a brand new service or adhoc (batch) job
>> created. The JobDescription is populated from the .aurora config. You may
>> want to trace "aurora job create" client command implementation to see how
>> it happens.
>>
>> 4. What is the Lock object? I see that some APIs require locking and some
>>> don't. For example, createJob needs a Lock object as parameter, & I am
>>> assuming that it is required so that one does not create multiple jobs with
>>> the same JobKey.
>>
>> Ignore this object as it's an echo of the old client updater. It's now
>> deprecated and will be removed soon. You can pass NULL for now.
>>
>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>
>> Another echo of the client updater but this time it's got a second life.
>> Check out its new signature and comments in the api.thrift. It's
>> essentially a "scale-out" API that can add instances to the existing job
>> without changing the underlying task assumptions.
>>
>> 6. getPendingResult(TaskQuery taskquery):
>>
>> It's actually 'getPendingReason' and is currently used exclusively by the
>> UI to get the reason for a task PENDING state.
>>
>> 7. setQuota & getQuota for setting user level quotas.
>>
>> This is to set role-level quota. Currently only required for tasks with
>> 'production=True'. Search through our docs for more details.
>>
>> 8. killTasks to kill all running instances of a job in the cluster.
>>
>> It's quite versatile and can be used to kill some or all instances of the
>> job.
>>
>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>
>> Your observations are correct. This is the main API to change your
>> service job in any way (including adding, removing or modifying instances).
>>
>> An aurora scheduling question is if I start a job with 5 instances, and
>>> there are resources available to run only 4 of them, does the entire job
>>> block, or only the 5th instance of the job blocks?
>>
>> Scheduler will try to schedule as many instances as it can. Those that
>> will not find resources will remain in PENDING state until more resources
>> are available. In your particular example only the 5th will remain PENDING.
>>
>>
>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com> wrote:
>>
>>> Hi,
>>> I was going through the Aurora Thrift API to determine how to add new
>>> jobs.
>>> I am using aurora v0.12 released last month and have upgraded to mesos
>>> v0.25 accordingly.
>>>
>>> Below is a summary of my (very limited) understanding of some APIs, &
>>> would help it if someone can point out flaws in my understanding:
>>>
>>> 1. All APIs require thrift inputs of the structs specified, and
>>> return thrift values only in Response.result field.
>>>
>>> 2. Is there a set of examples in the documentation to help
>>> understand Thrift API better?
>>>
>>> 3. createJob(JobDescription desc, Lock lock):
>>> This is basically the API to replace the Aurora DSL/.aurora files
>>> for job configuration.
>>>
>>> 4. What is the Lock object? I see that some APIs require locking and
>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>> assuming that it is required so that one does not create multiple jobs with
>>> the same JobKey.
>>>
>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>> By the naming convention, it seems this is used to increase the
>>> number of instances of a job. It will not result in stopping of current
>>> instances of the job.
>>>
>>> My second explanation for this API: Since it uses a set of
>>> instanceIds, this is used for adding already running job in slaves to the
>>> internal data structures of Aurora to track the job.
>>>
>>> 6. getPendingResult(TaskQuery taskquery):
>>> Return the reason (in string) about why the job is PENDING. For
>>> example: insufficient CPU.
>>>
>>> 7. setQuota & getQuota for setting user level quotas.
>>>
>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>
>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>> Used for updating jobs with the new TaskConfig specified. Can be
>>> used if resource requirement changes. For example: If I wanted aurora to
>>> update the version of container used for a job using TaskConfig.Container
>>> attribute.
>>>
>>>
>>> An aurora scheduling question is if I start a job with 5 instances, and
>>> there are resources available to run only 4 of them, does the entire job
>>> block, or only the 5th instance of the job blocks?
>>>
>>> Thanks!
>>>
>>> --
>>> κρισhναν
>>>
>>
>>
>
Re: Aurora Thrift API Info
Posted by Bill Farner <wf...@apache.org>.
If you'd rather not build from source, there's also a docker image for the
thrift compiler: https://hub.docker.com/r/thrift/thrift-compiler/
On Thu, Mar 17, 2016 at 9:32 AM, Krish <kr...@gmail.com> wrote:
> Jake/Chris,
> Thanks for the info.
> When I try to install thrift v0.9.3 from source, I get an error as follows
> while running `make check`:
> ...
> ...
> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 0.062 sec
> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 0.046 sec
>
> BUILD FAILED
> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>
> Total time: 17 seconds
> make[3]: *** [check-local] Error 1
> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
> make[1]: *** [check-recursive] Error 1
> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
> make: *** [check-recursive] Error 1
>
>
>
>
> --
> κρισhναν
>
> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <c....@gmail.com>
> wrote:
>
>> I've used the latest thrift to generate go code, and then manually
>> created executor config which works and is able to launch jobs.
>>
>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org> wrote:
>>
>>> Hi Krish
>>> We are using Thrift with go for all our api calls to Aurora, would
>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>
>>> happy to help answer any questions you might have
>>>
>>> -Jake
>>>
>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Thanks, Bill.
>>>>
>>>> Well I have started my foray into the the thrift API today. And I think
>>>> I am stuck with some thrift configs.
>>>>
>>>> Does it matter if I use thrift v0.9.0 on the client side to talk with
>>>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>> changelog or compatibility statement on the thrift project site.
>>>>
>>>>
>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos have
>>>> 0.9.0, I had to compile the thrift compiler v0.9.1 from source. However,
>>>> when I try to generate golang code, I think I hit a compiler bug:
>>>> krish@krish:/tmp
>>>> > thrift --gen go api.thrift
>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>
>>>> I can modify the golang code by hand, but I would like to play it safe
>>>> and use the working compiler from the debian repos.
>>>>
>>>> Also, when I use thrift v0.9.0, and try to integrate code into a test
>>>> golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>> similar error and gone past it?
>>>> I have already done a `go get
>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>>> wrote:
>>>>
>>>>> Regarding documentation - Maxim is correct that there isn't much in
>>>>> the way of independent/holistic docs for the thrift API. There is,
>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>
>>>>> If you are looking to use the thrift API directly, the most difficult
>>>>> API method will be defining the ExecutorConfig.data value when calling
>>>>> createJob. Please don't hesitate to ask for assistance if you get to that
>>>>> point!
>>>>>
>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>> return thrift values only in Response.result field.
>>>>>>
>>>>>> Correct. There is also 'details' field that may have additional
>>>>>> messages (of error or informational nature)
>>>>>>
>>>>>> 2. Is there a set of examples in the documentation to help understand
>>>>>>> Thrift API better?
>>>>>>
>>>>>> The thrift API is largely undocumented. There is an effort to bring
>>>>>> up a fully supported REST API that will presumably get documented and
>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>
>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>
>>>>>> This is the API to use when you a brand new service or adhoc (batch)
>>>>>> job created. The JobDescription is populated from the .aurora config. You
>>>>>> may want to trace "aurora job create" client command implementation to see
>>>>>> how it happens.
>>>>>>
>>>>>> 4. What is the Lock object? I see that some APIs require locking and
>>>>>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>>>>>> assuming that it is required so that one does not create multiple jobs with
>>>>>>> the same JobKey.
>>>>>>
>>>>>> Ignore this object as it's an echo of the old client updater. It's
>>>>>> now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>
>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>
>>>>>> Another echo of the client updater but this time it's got a second
>>>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>>>> essentially a "scale-out" API that can add instances to the existing job
>>>>>> without changing the underlying task assumptions.
>>>>>>
>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>
>>>>>> It's actually 'getPendingReason' and is currently used exclusively by
>>>>>> the UI to get the reason for a task PENDING state.
>>>>>>
>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>
>>>>>> This is to set role-level quota. Currently only required for tasks
>>>>>> with 'production=True'. Search through our docs for more details.
>>>>>>
>>>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>>>
>>>>>> It's quite versatile and can be used to kill some or all instances of
>>>>>> the job.
>>>>>>
>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>
>>>>>> Your observations are correct. This is the main API to change your
>>>>>> service job in any way (including adding, removing or modifying instances).
>>>>>>
>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>
>>>>>> Scheduler will try to schedule as many instances as it can. Those
>>>>>> that will not find resources will remain in PENDING state until more
>>>>>> resources are available. In your particular example only the 5th will
>>>>>> remain PENDING.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I was going through the Aurora Thrift API to determine how to add
>>>>>>> new jobs.
>>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>>> mesos v0.25 accordingly.
>>>>>>>
>>>>>>> Below is a summary of my (very limited) understanding of some APIs,
>>>>>>> & would help it if someone can point out flaws in my understanding:
>>>>>>>
>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>> return thrift values only in Response.result field.
>>>>>>>
>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>> understand Thrift API better?
>>>>>>>
>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>> files for job configuration.
>>>>>>>
>>>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>>>> with the same JobKey.
>>>>>>>
>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>> By the naming convention, it seems this is used to increase the
>>>>>>> number of instances of a job. It will not result in stopping of current
>>>>>>> instances of the job.
>>>>>>>
>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>
>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>> Return the reason (in string) about why the job is PENDING. For
>>>>>>> example: insufficient CPU.
>>>>>>>
>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>
>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>> cluster.
>>>>>>>
>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>> Used for updating jobs with the new TaskConfig specified. Can be
>>>>>>> used if resource requirement changes. For example: If I wanted aurora to
>>>>>>> update the version of container used for a job using TaskConfig.Container
>>>>>>> attribute.
>>>>>>>
>>>>>>>
>>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: Aurora Thrift API Info
Posted by Bill Farner <wf...@apache.org>.
Great to hear, onwards!
On Wednesday, March 23, 2016, Krish <kr...@gmail.com> wrote:
> Thanks Bill/wfarner, Jake/jfarrell, Steve/t3hSteve, Joshua/jcohen, Maxim &
> all the others (forgive me if I miss your name here), who took time out to
> guide me over mail & on IRC over the past few days.
>
> The last hurdle, till now that is, was this:
> The java source has a lot of parameters annotated with @Nullable, which
> means we can set this param to null. Accordingly, I was setting them to
> `nil` in the golang structs, which made the client break while accessing
> nil fields.
> I had to modify the generated code a bit (trivial changes), to make it
> write only the fields that were present.
>
> There is still a long way to go to actually call this 'done', however, it
> looks good so far; just implemented the addInstances & getPendingReason
> APIs in my thrift client, and it works flawlessly. Getting a hang of how
> things work.
>
> For all the folks who might be interested in playing with the thrift API,
> please have a look @https://github.com/krish7919/aurora_thrift_api/.
> It doesn't have great documentation, but some comments to go with source
> code.
> I am also going to put up a short writeup on aurora, and how to use the
> APIs in the coming few days.
>
>
>
> --
> κρισhναν
>
> On Tue, Mar 22, 2016 at 3:53 PM, Krish <krishnan.k.iyer@gmail.com
> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>> wrote:
>
>> Thanks, Bill! Program counter is moving forward! :)
>>
>> Since, golang net/http does not support redirects, I hacked my code to
>> query all aurora cluster members if the request is redirected. The proper
>> fix, as Jake Farrell rightly pointed out on IRC, would be to make the go
>> code handle redirects by using a properly configured http client. But I can
>> delay that for now for some time.
>>
>> I got the proper getJobs list. I parse the resultant response as:
>> //parse the response
>> y := resp.Result_.GetJobsResult_.Configs
>> fmt.Println("Configs: ", y)
>>
>> I am sure there is a better way to parse the response in golang, instead
>> of getting a map.
>> I killed all the mesos slaves & query for the pending reason for a job
>> (which was already scheduled), and have a few doubts about it.
>>
>> Firstly, I thought passing an empty TaskQuery structure is supposed to
>> give me all the pending reasons.
>>
>> Secondly, I see (at least in golang) the Owner field is mandatory. The
>> client fails with a null pointer reference if it's not present.
>>
>> Thirdly, once I add the Owner field, aurora client receives an error from
>> the server as below, even though I have not set the Statuses and SlaveHosts
>> field:
>>
>> > ./aurora_client_service --api "http://54.209.127.224:8081/api"
>> ResponseCode: 'INVALID_REQUEST'
>> ServerInfo: 'ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:})'
>> Details: '[ResponseDetail({Message:Statuses or slaveHosts are not
>> supported in TaskQuery(owner:Identity(role:, user:), role:, environment:,
>> jobName:my_job, taskIds:[], statuses:[], instanceIds:[], slaveHosts:[],
>> jobKeys:[], offset:0, limit:0)})]'
>> Result: '<nil>'
>>
>>
>> My task query struct is as follows:
>>
>> taskQuery := api.NewTaskQuery()
>> //taskQuery.JobName = "my_job"
>> //taskQuery.TaskIds = nil
>> //taskQuery.Statuses = nil
>> //taskQuery.InstanceIds = nil
>> //taskQuery.SlaveHosts = nil
>> //taskQuery.Environment = ""
>> //taskQuery.JobKeys = nil
>> //taskQuery.Offset = 0
>> //taskQuery.Limit = 0
>> //taskQuery.Role = ""
>> taskQuery.Owner = &api.Identity {
>> Role: "",
>> User: "",
>> }
>> resp, err := client.GetPendingReason(taskQuery)
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Tue, Mar 22, 2016 at 6:26 AM, Bill Farner <wfarner@apache.org
>> <javascript:_e(%7B%7D,'cvml','wfarner@apache.org');>> wrote:
>>
>>> Figured it out. I was first tipped off that the request being made is a
>>> GET, while i've normally seen HTTP thrift clients making POST requests. I
>>> haven't yet drilled into the rationale behind the alternative client in the
>>> thrift library, but this fixed up your code on my trials:
>>>
>>> - transport, err = thrift.NewTHttpClient(endpoint)
>>> + transport, err = thrift.NewTHttpPostClient(endpoint)
>>>
>>> On Sun, Mar 20, 2016 at 10:58 PM, Krish <krishnan.k.iyer@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>> wrote:
>>>
>>>> Thanks Bill.
>>>> I checked the HTTP request. The sent packet data is:
>>>> GET /api HTTP/1.1
>>>> Host: 54.209.127.223:8081
>>>> User-Agent: Go-http-client/1.1
>>>> Accept-Encoding: gzip
>>>>
>>>> I can attach the complete capture if that is needed.
>>>>
>>>> Link to my code on github:
>>>> https://github.com/krish7919/aurora_thrift_api
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <wfarner@apache.org
>>>> <javascript:_e(%7B%7D,'cvml','wfarner@apache.org');>> wrote:
>>>>
>>>>> Looks to me like it's failing before the request is even deserialized
>>>>> - in the thrift layer (the exception is thrown here
>>>>> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132>
>>>>> - EOF). Can you capture the HTTP request that was sent by the client?
>>>>> That might provide more clues.
>>>>>
>>>>> Also, if you're able to put up a git repo with your client code, i can
>>>>> poke at it.
>>>>>
>>>>> On Sat, Mar 19, 2016 at 11:26 AM, Krish <krishnan.k.iyer@gmail.com
>>>>> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>> wrote:
>>>>>
>>>>>> Hi Bill,
>>>>>> Tried digging more about aurora thrift API this weekend.
>>>>>> The thrift generated code is a good reference point.
>>>>>>
>>>>>> You are right; the '/' path just gives me the list of URLs that one
>>>>>> gets on the browser when I query it using my thrift client.
>>>>>> Any pointers based on the data below will be very helpful to find out
>>>>>> why is aurora bailing out when processing the thrift request, or is it some
>>>>>> client side error that is causing it.
>>>>>>
>>>>>>
>>>>>> My thrift client trying to query for getJobs:
>>>>>>
>>>>>> ....
>>>>>> var protocolFactory thrift.TProtocolFactory
>>>>>> var transport thrift.TTransport
>>>>>> var client *api.ReadOnlySchedulerClient
>>>>>> var err error
>>>>>> transport, err = thrift.NewTHttpClient("
>>>>>> http://54.209.17.221:8081/api")
>>>>>> defer transport.Close()
>>>>>> protocolFactory = thrift.NewTJSONProtocolFactory()
>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>>> protocolFactory)
>>>>>> err = transport.Open()
>>>>>> if err != nil {
>>>>>> fmt.Println("Error opening socket: ", err)
>>>>>> os.Exit(1)
>>>>>> }
>>>>>> defer transport.Close()
>>>>>> fmt.Println(client.GetJobs(""))
>>>>>> ....
>>>>>>
>>>>>>
>>>>>> I did a wireshark analysis of outgoing packets to aurora, & I do get
>>>>>> a response packet from aurora, and my thrift client bails out with an error
>>>>>> (runtime error: invalid memory address or nil pointer dereference). The
>>>>>> data in the packet sent by server is:
>>>>>>
>>>>>> <html>
>>>>>> <head>
>>>>>> <meta http-equiv="Content-Type"
>>>>>> content="text/html;charset=ISO-8859-1"/>
>>>>>> <title>Error 500 </title>
>>>>>> </head>
>>>>>> <body>
>>>>>> <h2>HTTP ERROR: 500</h2>
>>>>>> <p>Problem accessing /api. Reason:
>>>>>> <pre> javax.servlet.ServletException:
>>>>>> org.apache.thrift.transport.TTransportException</pre></p>
>>>>>> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
>>>>>> 9.3.6.v20151106</a><hr/>
>>>>>> </body>
>>>>>> </html>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Stacktrace from the server/aurora console logs:
>>>>>>
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
>>>>>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
>>>>>> org.apache.thrift.transport.TTransportException
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>>>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
>>>>>> ~[aurora-0.12.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>>>> ~[aurora-0.12.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
>>>>>> ~[aurora-0.12.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>>>> ~[aurora-0.12.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>>>> ~[guice-servlet-3.0.jar:na]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>>>>>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
>>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
>>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>>> ...
>>>>>> ...
>>>>>> ...
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
>>>>>> org.apache.thrift.transport.TTransportException: null
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames
>>>>>> omitted
>>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
>>>>>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
>>>>>> [19/Mar/2016:18:12:23 +
>>>>>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> κρισhναν
>>>>>>
>>>>>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <wfarner@apache.org
>>>>>> <javascript:_e(%7B%7D,'cvml','wfarner@apache.org');>> wrote:
>>>>>>
>>>>>>> 8081 is indeed the default thrift port. If you can capture a raw
>>>>>>> HTTP request, we could diagnose this better. My first hunch is that the
>>>>>>> thrift client expects the API to be mounted at /, when in fact we mount it
>>>>>>> at /api.
>>>>>>>
>>>>>>> Jake can probably tell you exactly what's wrong based on his code,
>>>>>>> but in the meantime you might find it helpful to compare against how we set
>>>>>>> up the JS and python clients:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <krishnan.k.iyer@gmail.com
>>>>>>> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>> wrote:
>>>>>>>
>>>>>>>> Apologies for multiple mails, the previous email was sent
>>>>>>>> accidentally.
>>>>>>>> I didn't add a problem description, before hitting send.
>>>>>>>>
>>>>>>>> When I query aurora for all the jobs using getJobs, I find the
>>>>>>>> aurora error as given below and the response I get using my client.
>>>>>>>>
>>>>>>>> --
>>>>>>>> κρισhναν
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <krishnan.k.iyer@gmail.com
>>>>>>>> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Jake!
>>>>>>>>>
>>>>>>>>> That worked like a charm, & I was wondering why does install from
>>>>>>>>> source doesn't work!
>>>>>>>>>
>>>>>>>>> Aurora Log:
>>>>>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318
>>>>>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>>>>>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>>>>>>>
>>>>>>>>> Thrift client:
>>>>>>>>> ./mrfantastic_service.out
>>>>>>>>> Connecting to aurora....
>>>>>>>>> <nil> EOF
>>>>>>>>>
>>>>>>>>> Thrift client sourcec:
>>>>>>>>> func thriftAuroraJobs() {
>>>>>>>>> var protocolFactory thrift.TProtocolFactory
>>>>>>>>> var transportFactory thrift.TTransportFactory
>>>>>>>>> var transport thrift.TTransport
>>>>>>>>> var client *api.ReadOnlySchedulerClient
>>>>>>>>> var err error
>>>>>>>>>
>>>>>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>>>>>>>> //transportFactory =
>>>>>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>>>>>>>> transportFactory = thrift.NewTTransportFactory()
>>>>>>>>> fmt.Println("Connecting to aurora....")
>>>>>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>>>>>>>> if err != nil {
>>>>>>>>> fmt.Println("Error opening socket:", err)
>>>>>>>>> os.Exit(1)
>>>>>>>>> //return err
>>>>>>>>> }
>>>>>>>>> if transport == nil {
>>>>>>>>> os.Exit(1)
>>>>>>>>> }
>>>>>>>>> transport = transportFactory.GetTransport(transport)
>>>>>>>>> if transport == nil {
>>>>>>>>> os.Exit(1)
>>>>>>>>> }
>>>>>>>>> err = transport.Open()
>>>>>>>>> if err != nil {
>>>>>>>>> os.Exit(1)
>>>>>>>>> }
>>>>>>>>> defer transport.Close()
>>>>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>>>>>> protocolFactory)
>>>>>>>>> fmt.Println(client.GetJobs(""))
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora
>>>>>>>>> is looking for HTTP data on the socket.
>>>>>>>>> Is there a config that needs to be set for thrift API to be
>>>>>>>>> initialized?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> κρισhναν
>>>>>>>>>
>>>>>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <
>>>>>>>>> jfarrell@apache.org
>>>>>>>>> <javascript:_e(%7B%7D,'cvml','jfarrell@apache.org');>> wrote:
>>>>>>>>>
>>>>>>>>>> if you just need the compiler you can install the thrift-compiler
>>>>>>>>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>>>>>>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>>>>>>>
>>>>>>>>>> -Jake
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> deb packaging (tested with ubuntu trusty)
>>>>>>>>>>
>>>>>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>>>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>>>>>>>> > /etc/apt/sources.list.d/thrift.list
>>>>>>>>>>
>>>>>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> or for centos/rhel (tested with centos 7.2)
>>>>>>>>>>
>>>>>>>>>> > /etc/yum.repos.d/thrift.repo
>>>>>>>>>>
>>>>>>>>>> [thrift]
>>>>>>>>>> name=Apache Thrift rpm repo
>>>>>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>>>>>>>> enabled=1
>>>>>>>>>> gpgcheck=0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <
>>>>>>>>>> krishnan.k.iyer@gmail.com
>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Jake/Chris,
>>>>>>>>>>> Thanks for the info.
>>>>>>>>>>> When I try to install thrift v0.9.3 from source, I get an error
>>>>>>>>>>> as follows while running `make check`:
>>>>>>>>>>> ...
>>>>>>>>>>> ...
>>>>>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0,
>>>>>>>>>>> Time elapsed: 0.062 sec
>>>>>>>>>>> [junit] Running
>>>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0,
>>>>>>>>>>> Time elapsed: 0.046 sec
>>>>>>>>>>>
>>>>>>>>>>> BUILD FAILED
>>>>>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>>>>>>>
>>>>>>>>>>> Total time: 17 seconds
>>>>>>>>>>> make[3]: *** [check-local] Error 1
>>>>>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>>>> make[2]: *** [check-am] Error 2
>>>>>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>>>> make[1]: *** [check-recursive] Error 1
>>>>>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>>>>>>>> make: *** [check-recursive] Error 1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> κρισhναν
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>>>>>>>> c.bannister@gmail.com
>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','c.bannister@gmail.com');>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I've used the latest thrift to generate go code, and then
>>>>>>>>>>>> manually created executor config which works and is able to launch jobs.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <
>>>>>>>>>>>> jfarrell@apache.org
>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','jfarrell@apache.org');>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Krish
>>>>>>>>>>>>> We are using Thrift with go for all our api calls to Aurora,
>>>>>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>>>>>>>
>>>>>>>>>>>>> happy to help answer any questions you might have
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Jake
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <
>>>>>>>>>>>>> krishnan.k.iyer@gmail.com
>>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, Bill.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Well I have started my foray into the the thrift API today.
>>>>>>>>>>>>>> And I think I am stuck with some thrift configs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to
>>>>>>>>>>>>>> talk with aurora using thrift 0.9.1? Are they compatible? I couldn't find
>>>>>>>>>>>>>> any changelog or compatibility statement on the thrift project site.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian
>>>>>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>>>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>>>>>>>>> krish@krish:/tmp
>>>>>>>>>>>>>> > thrift --gen go api.thrift
>>>>>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite
>>>>>>>>>>>>>> literal
>>>>>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I can modify the golang code by hand, but I would like to
>>>>>>>>>>>>>> play it safe and use the working compiler from the debian repos.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code
>>>>>>>>>>>>>> into a test golang app, it fails to find "thriftlib/api" package. Anyone
>>>>>>>>>>>>>> faced a similar error and gone past it?
>>>>>>>>>>>>>> I have already done a `go get
>>>>>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <
>>>>>>>>>>>>>> wfarner@apache.org
>>>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','wfarner@apache.org');>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't
>>>>>>>>>>>>>>> much in the way of independent/holistic docs for the thrift API. There is,
>>>>>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>>>>>>>>> to that point!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>>>>>>>> maxim@apache.org
>>>>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','maxim@apache.org');>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Correct. There is also 'details' field that may have
>>>>>>>>>>>>>>>> additional messages (of error or informational nature)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort
>>>>>>>>>>>>>>>> to bring up a fully supported REST API that will presumably get documented
>>>>>>>>>>>>>>>> and become much easier to use. It's mostly in flux now.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is the API to use when you a brand new service or
>>>>>>>>>>>>>>>> adhoc (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ignore this object as it's an echo of the old client
>>>>>>>>>>>>>>>> updater. It's now deprecated and will be removed soon. You can pass NULL
>>>>>>>>>>>>>>>> for now.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>>>>>>>> second life. Check out its new signature and comments in the api.thrift.
>>>>>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the existing
>>>>>>>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is to set role-level quota. Currently only required
>>>>>>>>>>>>>>>> for tasks with 'production=True'. Search through our docs for more details.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>>>>>>> cluster.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>>>>>>>> instances of the job.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string
>>>>>>>>>>>>>>>>> message):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your observations are correct. This is the main API to
>>>>>>>>>>>>>>>> change your service job in any way (including adding, removing or modifying
>>>>>>>>>>>>>>>> instances).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can.
>>>>>>>>>>>>>>>> Those that will not find resources will remain in PENDING state until more
>>>>>>>>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>>>>>>>>> remain PENDING.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <
>>>>>>>>>>>>>>>> krishnan.k.iyer@gmail.com
>>>>>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','krishnan.k.iyer@gmail.com');>
>>>>>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how
>>>>>>>>>>>>>>>>> to add new jobs.
>>>>>>>>>>>>>>>>> I am using aurora v0.12 released last month and have
>>>>>>>>>>>>>>>>> upgraded to mesos v0.25 accordingly.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of
>>>>>>>>>>>>>>>>> some APIs, & would help it if someone can point out flaws in my
>>>>>>>>>>>>>>>>> understanding:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs
>>>>>>>>>>>>>>>>> specified, and return thrift values only in Response.result field.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to
>>>>>>>>>>>>>>>>> help understand Thrift API better?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>>>> This is basically the API to replace the Aurora
>>>>>>>>>>>>>>>>> DSL/.aurora files for job configuration.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs
>>>>>>>>>>>>>>>>> require locking and some don't. For example, createJob needs a Lock object
>>>>>>>>>>>>>>>>> as parameter, & I am assuming that it is required so that one does not
>>>>>>>>>>>>>>>>> create multiple jobs with the same JobKey.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>>>> By the naming convention, it seems this is used to
>>>>>>>>>>>>>>>>> increase the number of instances of a job. It will not result in stopping
>>>>>>>>>>>>>>>>> of current instances of the job.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My second explanation for this API: Since it uses a
>>>>>>>>>>>>>>>>> set of instanceIds, this is used for adding already running job in slaves
>>>>>>>>>>>>>>>>> to the internal data structures of Aurora to track the job.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>>>> Return the reason (in string) about why the job is
>>>>>>>>>>>>>>>>> PENDING. For example: insufficient CPU.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in
>>>>>>>>>>>>>>>>> the cluster.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string
>>>>>>>>>>>>>>>>> message):
>>>>>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig
>>>>>>>>>>>>>>>>> specified. Can be used if resource requirement changes. For example: If I
>>>>>>>>>>>>>>>>> wanted aurora to update the version of container used for a job using
>>>>>>>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Thanks Bill/wfarner, Jake/jfarrell, Steve/t3hSteve, Joshua/jcohen, Maxim &
all the others (forgive me if I miss your name here), who took time out to
guide me over mail & on IRC over the past few days.
The last hurdle, till now that is, was this:
The java source has a lot of parameters annotated with @Nullable, which
means we can set this param to null. Accordingly, I was setting them to
`nil` in the golang structs, which made the client break while accessing
nil fields.
I had to modify the generated code a bit (trivial changes), to make it
write only the fields that were present.
There is still a long way to go to actually call this 'done', however, it
looks good so far; just implemented the addInstances & getPendingReason
APIs in my thrift client, and it works flawlessly. Getting a hang of how
things work.
For all the folks who might be interested in playing with the thrift API,
please have a look @https://github.com/krish7919/aurora_thrift_api/.
It doesn't have great documentation, but some comments to go with source
code.
I am also going to put up a short writeup on aurora, and how to use the
APIs in the coming few days.
--
κρισhναν
On Tue, Mar 22, 2016 at 3:53 PM, Krish <kr...@gmail.com> wrote:
> Thanks, Bill! Program counter is moving forward! :)
>
> Since, golang net/http does not support redirects, I hacked my code to
> query all aurora cluster members if the request is redirected. The proper
> fix, as Jake Farrell rightly pointed out on IRC, would be to make the go
> code handle redirects by using a properly configured http client. But I can
> delay that for now for some time.
>
> I got the proper getJobs list. I parse the resultant response as:
> //parse the response
> y := resp.Result_.GetJobsResult_.Configs
> fmt.Println("Configs: ", y)
>
> I am sure there is a better way to parse the response in golang, instead
> of getting a map.
> I killed all the mesos slaves & query for the pending reason for a job
> (which was already scheduled), and have a few doubts about it.
>
> Firstly, I thought passing an empty TaskQuery structure is supposed to
> give me all the pending reasons.
>
> Secondly, I see (at least in golang) the Owner field is mandatory. The
> client fails with a null pointer reference if it's not present.
>
> Thirdly, once I add the Owner field, aurora client receives an error from
> the server as below, even though I have not set the Statuses and SlaveHosts
> field:
>
> > ./aurora_client_service --api "http://54.209.127.224:8081/api"
> ResponseCode: 'INVALID_REQUEST'
> ServerInfo: 'ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:})'
> Details: '[ResponseDetail({Message:Statuses or slaveHosts are not
> supported in TaskQuery(owner:Identity(role:, user:), role:, environment:,
> jobName:my_job, taskIds:[], statuses:[], instanceIds:[], slaveHosts:[],
> jobKeys:[], offset:0, limit:0)})]'
> Result: '<nil>'
>
>
> My task query struct is as follows:
>
> taskQuery := api.NewTaskQuery()
> //taskQuery.JobName = "my_job"
> //taskQuery.TaskIds = nil
> //taskQuery.Statuses = nil
> //taskQuery.InstanceIds = nil
> //taskQuery.SlaveHosts = nil
> //taskQuery.Environment = ""
> //taskQuery.JobKeys = nil
> //taskQuery.Offset = 0
> //taskQuery.Limit = 0
> //taskQuery.Role = ""
> taskQuery.Owner = &api.Identity {
> Role: "",
> User: "",
> }
> resp, err := client.GetPendingReason(taskQuery)
>
>
>
>
> --
> κρισhναν
>
> On Tue, Mar 22, 2016 at 6:26 AM, Bill Farner <wf...@apache.org> wrote:
>
>> Figured it out. I was first tipped off that the request being made is a
>> GET, while i've normally seen HTTP thrift clients making POST requests. I
>> haven't yet drilled into the rationale behind the alternative client in the
>> thrift library, but this fixed up your code on my trials:
>>
>> - transport, err = thrift.NewTHttpClient(endpoint)
>> + transport, err = thrift.NewTHttpPostClient(endpoint)
>>
>> On Sun, Mar 20, 2016 at 10:58 PM, Krish <kr...@gmail.com>
>> wrote:
>>
>>> Thanks Bill.
>>> I checked the HTTP request. The sent packet data is:
>>> GET /api HTTP/1.1
>>> Host: 54.209.127.223:8081
>>> User-Agent: Go-http-client/1.1
>>> Accept-Encoding: gzip
>>>
>>> I can attach the complete capture if that is needed.
>>>
>>> Link to my code on github:
>>> https://github.com/krish7919/aurora_thrift_api
>>>
>>>
>>>
>>>
>>>
>>> --
>>> κρισhναν
>>>
>>> On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <wf...@apache.org> wrote:
>>>
>>>> Looks to me like it's failing before the request is even deserialized -
>>>> in the thrift layer (the exception is thrown here
>>>> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132>
>>>> - EOF). Can you capture the HTTP request that was sent by the client?
>>>> That might provide more clues.
>>>>
>>>> Also, if you're able to put up a git repo with your client code, i can
>>>> poke at it.
>>>>
>>>> On Sat, Mar 19, 2016 at 11:26 AM, Krish <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Bill,
>>>>> Tried digging more about aurora thrift API this weekend.
>>>>> The thrift generated code is a good reference point.
>>>>>
>>>>> You are right; the '/' path just gives me the list of URLs that one
>>>>> gets on the browser when I query it using my thrift client.
>>>>> Any pointers based on the data below will be very helpful to find out
>>>>> why is aurora bailing out when processing the thrift request, or is it some
>>>>> client side error that is causing it.
>>>>>
>>>>>
>>>>> My thrift client trying to query for getJobs:
>>>>>
>>>>> ....
>>>>> var protocolFactory thrift.TProtocolFactory
>>>>> var transport thrift.TTransport
>>>>> var client *api.ReadOnlySchedulerClient
>>>>> var err error
>>>>> transport, err = thrift.NewTHttpClient("
>>>>> http://54.209.17.221:8081/api")
>>>>> defer transport.Close()
>>>>> protocolFactory = thrift.NewTJSONProtocolFactory()
>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>> protocolFactory)
>>>>> err = transport.Open()
>>>>> if err != nil {
>>>>> fmt.Println("Error opening socket: ", err)
>>>>> os.Exit(1)
>>>>> }
>>>>> defer transport.Close()
>>>>> fmt.Println(client.GetJobs(""))
>>>>> ....
>>>>>
>>>>>
>>>>> I did a wireshark analysis of outgoing packets to aurora, & I do get a
>>>>> response packet from aurora, and my thrift client bails out with an error
>>>>> (runtime error: invalid memory address or nil pointer dereference). The
>>>>> data in the packet sent by server is:
>>>>>
>>>>> <html>
>>>>> <head>
>>>>> <meta http-equiv="Content-Type"
>>>>> content="text/html;charset=ISO-8859-1"/>
>>>>> <title>Error 500 </title>
>>>>> </head>
>>>>> <body>
>>>>> <h2>HTTP ERROR: 500</h2>
>>>>> <p>Problem accessing /api. Reason:
>>>>> <pre> javax.servlet.ServletException:
>>>>> org.apache.thrift.transport.TTransportException</pre></p>
>>>>> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
>>>>> 9.3.6.v20151106</a><hr/>
>>>>> </body>
>>>>> </html>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Stacktrace from the server/aurora console logs:
>>>>>
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
>>>>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
>>>>> org.apache.thrift.transport.TTransportException
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
>>>>> ~[aurora-0.12.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>>> ~[aurora-0.12.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
>>>>> ~[aurora-0.12.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>>> ~[aurora-0.12.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>>> ~[guice-servlet-3.0.jar:na]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>>>>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>>> ...
>>>>> ...
>>>>> ...
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
>>>>> org.apache.thrift.transport.TTransportException: null
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames
>>>>> omitted
>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
>>>>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
>>>>> [19/Mar/2016:18:12:23 +
>>>>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <wf...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> 8081 is indeed the default thrift port. If you can capture a raw
>>>>>> HTTP request, we could diagnose this better. My first hunch is that the
>>>>>> thrift client expects the API to be mounted at /, when in fact we mount it
>>>>>> at /api.
>>>>>>
>>>>>> Jake can probably tell you exactly what's wrong based on his code,
>>>>>> but in the meantime you might find it helpful to compare against how we set
>>>>>> up the JS and python clients:
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Apologies for multiple mails, the previous email was sent
>>>>>>> accidentally.
>>>>>>> I didn't add a problem description, before hitting send.
>>>>>>>
>>>>>>> When I query aurora for all the jobs using getJobs, I find the
>>>>>>> aurora error as given below and the response I get using my client.
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Jake!
>>>>>>>>
>>>>>>>> That worked like a charm, & I was wondering why does install from
>>>>>>>> source doesn't work!
>>>>>>>>
>>>>>>>> Aurora Log:
>>>>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318
>>>>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>>>>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>>>>>>
>>>>>>>> Thrift client:
>>>>>>>> ./mrfantastic_service.out
>>>>>>>> Connecting to aurora....
>>>>>>>> <nil> EOF
>>>>>>>>
>>>>>>>> Thrift client sourcec:
>>>>>>>> func thriftAuroraJobs() {
>>>>>>>> var protocolFactory thrift.TProtocolFactory
>>>>>>>> var transportFactory thrift.TTransportFactory
>>>>>>>> var transport thrift.TTransport
>>>>>>>> var client *api.ReadOnlySchedulerClient
>>>>>>>> var err error
>>>>>>>>
>>>>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>>>>>>> //transportFactory =
>>>>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>>>>>>> transportFactory = thrift.NewTTransportFactory()
>>>>>>>> fmt.Println("Connecting to aurora....")
>>>>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>>>>>>> if err != nil {
>>>>>>>> fmt.Println("Error opening socket:", err)
>>>>>>>> os.Exit(1)
>>>>>>>> //return err
>>>>>>>> }
>>>>>>>> if transport == nil {
>>>>>>>> os.Exit(1)
>>>>>>>> }
>>>>>>>> transport = transportFactory.GetTransport(transport)
>>>>>>>> if transport == nil {
>>>>>>>> os.Exit(1)
>>>>>>>> }
>>>>>>>> err = transport.Open()
>>>>>>>> if err != nil {
>>>>>>>> os.Exit(1)
>>>>>>>> }
>>>>>>>> defer transport.Close()
>>>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>>>>> protocolFactory)
>>>>>>>> fmt.Println(client.GetJobs(""))
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora
>>>>>>>> is looking for HTTP data on the socket.
>>>>>>>> Is there a config that needs to be set for thrift API to be
>>>>>>>> initialized?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> κρισhναν
>>>>>>>>
>>>>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jfarrell@apache.org
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> if you just need the compiler you can install the thrift-compiler
>>>>>>>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>>>>>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>>>>>>
>>>>>>>>> -Jake
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> deb packaging (tested with ubuntu trusty)
>>>>>>>>>
>>>>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>>>>>>> > /etc/apt/sources.list.d/thrift.list
>>>>>>>>>
>>>>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> or for centos/rhel (tested with centos 7.2)
>>>>>>>>>
>>>>>>>>> > /etc/yum.repos.d/thrift.repo
>>>>>>>>>
>>>>>>>>> [thrift]
>>>>>>>>> name=Apache Thrift rpm repo
>>>>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>>>>>>> enabled=1
>>>>>>>>> gpgcheck=0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <krishnan.k.iyer@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Jake/Chris,
>>>>>>>>>> Thanks for the info.
>>>>>>>>>> When I try to install thrift v0.9.3 from source, I get an error
>>>>>>>>>> as follows while running `make check`:
>>>>>>>>>> ...
>>>>>>>>>> ...
>>>>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0,
>>>>>>>>>> Time elapsed: 0.062 sec
>>>>>>>>>> [junit] Running
>>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0,
>>>>>>>>>> Time elapsed: 0.046 sec
>>>>>>>>>>
>>>>>>>>>> BUILD FAILED
>>>>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>>>>>>
>>>>>>>>>> Total time: 17 seconds
>>>>>>>>>> make[3]: *** [check-local] Error 1
>>>>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>>> make[2]: *** [check-am] Error 2
>>>>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>>> make[1]: *** [check-recursive] Error 1
>>>>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>>>>>>> make: *** [check-recursive] Error 1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> κρισhναν
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>>>>>>> c.bannister@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I've used the latest thrift to generate go code, and then
>>>>>>>>>>> manually created executor config which works and is able to launch jobs.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <
>>>>>>>>>>> jfarrell@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Krish
>>>>>>>>>>>> We are using Thrift with go for all our api calls to Aurora,
>>>>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>>>>>>
>>>>>>>>>>>> happy to help answer any questions you might have
>>>>>>>>>>>>
>>>>>>>>>>>> -Jake
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <
>>>>>>>>>>>> krishnan.k.iyer@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, Bill.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Well I have started my foray into the the thrift API today.
>>>>>>>>>>>>> And I think I am stuck with some thrift configs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to
>>>>>>>>>>>>> talk with aurora using thrift 0.9.1? Are they compatible? I couldn't find
>>>>>>>>>>>>> any changelog or compatibility statement on the thrift project site.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian
>>>>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>>>>>>>> krish@krish:/tmp
>>>>>>>>>>>>> > thrift --gen go api.thrift
>>>>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite
>>>>>>>>>>>>> literal
>>>>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can modify the golang code by hand, but I would like to play
>>>>>>>>>>>>> it safe and use the working compiler from the debian repos.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into
>>>>>>>>>>>>> a test golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>>>>>>>>> similar error and gone past it?
>>>>>>>>>>>>> I have already done a `go get
>>>>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <
>>>>>>>>>>>>> wfarner@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't
>>>>>>>>>>>>>> much in the way of independent/holistic docs for the thrift API. There is,
>>>>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>>>>>>>> to that point!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>>>>>>> maxim@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Correct. There is also 'details' field that may have
>>>>>>>>>>>>>>> additional messages (of error or informational nature)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort
>>>>>>>>>>>>>>> to bring up a fully supported REST API that will presumably get documented
>>>>>>>>>>>>>>> and become much easier to use. It's mostly in flux now.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ignore this object as it's an echo of the old client
>>>>>>>>>>>>>>> updater. It's now deprecated and will be removed soon. You can pass NULL
>>>>>>>>>>>>>>> for now.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>>>>>>> second life. Check out its new signature and comments in the api.thrift.
>>>>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the existing
>>>>>>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is to set role-level quota. Currently only required for
>>>>>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more details.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>>>>>> cluster.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>>>>>>> instances of the job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your observations are correct. This is the main API to
>>>>>>>>>>>>>>> change your service job in any way (including adding, removing or modifying
>>>>>>>>>>>>>>> instances).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can.
>>>>>>>>>>>>>>> Those that will not find resources will remain in PENDING state until more
>>>>>>>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>>>>>>>> remain PENDING.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <
>>>>>>>>>>>>>>> krishnan.k.iyer@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how
>>>>>>>>>>>>>>>> to add new jobs.
>>>>>>>>>>>>>>>> I am using aurora v0.12 released last month and have
>>>>>>>>>>>>>>>> upgraded to mesos v0.25 accordingly.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of
>>>>>>>>>>>>>>>> some APIs, & would help it if someone can point out flaws in my
>>>>>>>>>>>>>>>> understanding:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs
>>>>>>>>>>>>>>>> specified, and return thrift values only in Response.result field.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to
>>>>>>>>>>>>>>>> help understand Thrift API better?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>>> This is basically the API to replace the Aurora
>>>>>>>>>>>>>>>> DSL/.aurora files for job configuration.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs
>>>>>>>>>>>>>>>> require locking and some don't. For example, createJob needs a Lock object
>>>>>>>>>>>>>>>> as parameter, & I am assuming that it is required so that one does not
>>>>>>>>>>>>>>>> create multiple jobs with the same JobKey.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>>> By the naming convention, it seems this is used to
>>>>>>>>>>>>>>>> increase the number of instances of a job. It will not result in stopping
>>>>>>>>>>>>>>>> of current instances of the job.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My second explanation for this API: Since it uses a set
>>>>>>>>>>>>>>>> of instanceIds, this is used for adding already running job in slaves to
>>>>>>>>>>>>>>>> the internal data structures of Aurora to track the job.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>>> Return the reason (in string) about why the job is
>>>>>>>>>>>>>>>> PENDING. For example: insufficient CPU.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in
>>>>>>>>>>>>>>>> the cluster.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string
>>>>>>>>>>>>>>>> message):
>>>>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig
>>>>>>>>>>>>>>>> specified. Can be used if resource requirement changes. For example: If I
>>>>>>>>>>>>>>>> wanted aurora to update the version of container used for a job using
>>>>>>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Thanks, Bill! Program counter is moving forward! :)
Since, golang net/http does not support redirects, I hacked my code to
query all aurora cluster members if the request is redirected. The proper
fix, as Jake Farrell rightly pointed out on IRC, would be to make the go
code handle redirects by using a properly configured http client. But I can
delay that for now for some time.
I got the proper getJobs list. I parse the resultant response as:
//parse the response
y := resp.Result_.GetJobsResult_.Configs
fmt.Println("Configs: ", y)
I am sure there is a better way to parse the response in golang, instead of
getting a map.
I killed all the mesos slaves & query for the pending reason for a job
(which was already scheduled), and have a few doubts about it.
Firstly, I thought passing an empty TaskQuery structure is supposed to give
me all the pending reasons.
Secondly, I see (at least in golang) the Owner field is mandatory. The
client fails with a null pointer reference if it's not present.
Thirdly, once I add the Owner field, aurora client receives an error from
the server as below, even though I have not set the Statuses and SlaveHosts
field:
> ./aurora_client_service --api "http://54.209.127.224:8081/api"
ResponseCode: 'INVALID_REQUEST'
ServerInfo: 'ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:})'
Details: '[ResponseDetail({Message:Statuses or slaveHosts are not supported
in TaskQuery(owner:Identity(role:, user:), role:, environment:,
jobName:my_job, taskIds:[], statuses:[], instanceIds:[], slaveHosts:[],
jobKeys:[], offset:0, limit:0)})]'
Result: '<nil>'
My task query struct is as follows:
taskQuery := api.NewTaskQuery()
//taskQuery.JobName = "my_job"
//taskQuery.TaskIds = nil
//taskQuery.Statuses = nil
//taskQuery.InstanceIds = nil
//taskQuery.SlaveHosts = nil
//taskQuery.Environment = ""
//taskQuery.JobKeys = nil
//taskQuery.Offset = 0
//taskQuery.Limit = 0
//taskQuery.Role = ""
taskQuery.Owner = &api.Identity {
Role: "",
User: "",
}
resp, err := client.GetPendingReason(taskQuery)
--
κρισhναν
On Tue, Mar 22, 2016 at 6:26 AM, Bill Farner <wf...@apache.org> wrote:
> Figured it out. I was first tipped off that the request being made is a
> GET, while i've normally seen HTTP thrift clients making POST requests. I
> haven't yet drilled into the rationale behind the alternative client in the
> thrift library, but this fixed up your code on my trials:
>
> - transport, err = thrift.NewTHttpClient(endpoint)
> + transport, err = thrift.NewTHttpPostClient(endpoint)
>
> On Sun, Mar 20, 2016 at 10:58 PM, Krish <kr...@gmail.com> wrote:
>
>> Thanks Bill.
>> I checked the HTTP request. The sent packet data is:
>> GET /api HTTP/1.1
>> Host: 54.209.127.223:8081
>> User-Agent: Go-http-client/1.1
>> Accept-Encoding: gzip
>>
>> I can attach the complete capture if that is needed.
>>
>> Link to my code on github:
>> https://github.com/krish7919/aurora_thrift_api
>>
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <wf...@apache.org> wrote:
>>
>>> Looks to me like it's failing before the request is even deserialized -
>>> in the thrift layer (the exception is thrown here
>>> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132>
>>> - EOF). Can you capture the HTTP request that was sent by the client?
>>> That might provide more clues.
>>>
>>> Also, if you're able to put up a git repo with your client code, i can
>>> poke at it.
>>>
>>> On Sat, Mar 19, 2016 at 11:26 AM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi Bill,
>>>> Tried digging more about aurora thrift API this weekend.
>>>> The thrift generated code is a good reference point.
>>>>
>>>> You are right; the '/' path just gives me the list of URLs that one
>>>> gets on the browser when I query it using my thrift client.
>>>> Any pointers based on the data below will be very helpful to find out
>>>> why is aurora bailing out when processing the thrift request, or is it some
>>>> client side error that is causing it.
>>>>
>>>>
>>>> My thrift client trying to query for getJobs:
>>>>
>>>> ....
>>>> var protocolFactory thrift.TProtocolFactory
>>>> var transport thrift.TTransport
>>>> var client *api.ReadOnlySchedulerClient
>>>> var err error
>>>> transport, err = thrift.NewTHttpClient("
>>>> http://54.209.17.221:8081/api")
>>>> defer transport.Close()
>>>> protocolFactory = thrift.NewTJSONProtocolFactory()
>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>> protocolFactory)
>>>> err = transport.Open()
>>>> if err != nil {
>>>> fmt.Println("Error opening socket: ", err)
>>>> os.Exit(1)
>>>> }
>>>> defer transport.Close()
>>>> fmt.Println(client.GetJobs(""))
>>>> ....
>>>>
>>>>
>>>> I did a wireshark analysis of outgoing packets to aurora, & I do get a
>>>> response packet from aurora, and my thrift client bails out with an error
>>>> (runtime error: invalid memory address or nil pointer dereference). The
>>>> data in the packet sent by server is:
>>>>
>>>> <html>
>>>> <head>
>>>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
>>>> <title>Error 500 </title>
>>>> </head>
>>>> <body>
>>>> <h2>HTTP ERROR: 500</h2>
>>>> <p>Problem accessing /api. Reason:
>>>> <pre> javax.servlet.ServletException:
>>>> org.apache.thrift.transport.TTransportException</pre></p>
>>>> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
>>>> 9.3.6.v20151106</a><hr/>
>>>> </body>
>>>> </html>
>>>>
>>>>
>>>>
>>>>
>>>> Stacktrace from the server/aurora console logs:
>>>>
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
>>>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
>>>> org.apache.thrift.transport.TTransportException
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>>>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> ...
>>>> ...
>>>> ...
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
>>>> org.apache.thrift.transport.TTransportException: null
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
>>>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
>>>> [19/Mar/2016:18:12:23 +
>>>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <wf...@apache.org>
>>>> wrote:
>>>>
>>>>> 8081 is indeed the default thrift port. If you can capture a raw HTTP
>>>>> request, we could diagnose this better. My first hunch is that the thrift
>>>>> client expects the API to be mounted at /, when in fact we mount it at /api.
>>>>>
>>>>> Jake can probably tell you exactly what's wrong based on his code, but
>>>>> in the meantime you might find it helpful to compare against how we set up
>>>>> the JS and python clients:
>>>>>
>>>>>
>>>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>>>>>
>>>>>
>>>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>>>>>
>>>>>
>>>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Apologies for multiple mails, the previous email was sent
>>>>>> accidentally.
>>>>>> I didn't add a problem description, before hitting send.
>>>>>>
>>>>>> When I query aurora for all the jobs using getJobs, I find the aurora
>>>>>> error as given below and the response I get using my client.
>>>>>>
>>>>>> --
>>>>>> κρισhναν
>>>>>>
>>>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Jake!
>>>>>>>
>>>>>>> That worked like a charm, & I was wondering why does install from
>>>>>>> source doesn't work!
>>>>>>>
>>>>>>> Aurora Log:
>>>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318
>>>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>>>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>>>>>
>>>>>>> Thrift client:
>>>>>>> ./mrfantastic_service.out
>>>>>>> Connecting to aurora....
>>>>>>> <nil> EOF
>>>>>>>
>>>>>>> Thrift client sourcec:
>>>>>>> func thriftAuroraJobs() {
>>>>>>> var protocolFactory thrift.TProtocolFactory
>>>>>>> var transportFactory thrift.TTransportFactory
>>>>>>> var transport thrift.TTransport
>>>>>>> var client *api.ReadOnlySchedulerClient
>>>>>>> var err error
>>>>>>>
>>>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>>>>>> //transportFactory =
>>>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>>>>>> transportFactory = thrift.NewTTransportFactory()
>>>>>>> fmt.Println("Connecting to aurora....")
>>>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>>>>>> if err != nil {
>>>>>>> fmt.Println("Error opening socket:", err)
>>>>>>> os.Exit(1)
>>>>>>> //return err
>>>>>>> }
>>>>>>> if transport == nil {
>>>>>>> os.Exit(1)
>>>>>>> }
>>>>>>> transport = transportFactory.GetTransport(transport)
>>>>>>> if transport == nil {
>>>>>>> os.Exit(1)
>>>>>>> }
>>>>>>> err = transport.Open()
>>>>>>> if err != nil {
>>>>>>> os.Exit(1)
>>>>>>> }
>>>>>>> defer transport.Close()
>>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>>>> protocolFactory)
>>>>>>> fmt.Println(client.GetJobs(""))
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora is
>>>>>>> looking for HTTP data on the socket.
>>>>>>> Is there a config that needs to be set for thrift API to be
>>>>>>> initialized?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> if you just need the compiler you can install the thrift-compiler
>>>>>>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>>>>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>>>>>
>>>>>>>> -Jake
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> deb packaging (tested with ubuntu trusty)
>>>>>>>>
>>>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>>>>>> > /etc/apt/sources.list.d/thrift.list
>>>>>>>>
>>>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>>>>>
>>>>>>>>
>>>>>>>> or for centos/rhel (tested with centos 7.2)
>>>>>>>>
>>>>>>>> > /etc/yum.repos.d/thrift.repo
>>>>>>>>
>>>>>>>> [thrift]
>>>>>>>> name=Apache Thrift rpm repo
>>>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>>>>>> enabled=1
>>>>>>>> gpgcheck=0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Jake/Chris,
>>>>>>>>> Thanks for the info.
>>>>>>>>> When I try to install thrift v0.9.3 from source, I get an error as
>>>>>>>>> follows while running `make check`:
>>>>>>>>> ...
>>>>>>>>> ...
>>>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>>>>>> elapsed: 0.062 sec
>>>>>>>>> [junit] Running
>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>>>>>>>> elapsed: 0.046 sec
>>>>>>>>>
>>>>>>>>> BUILD FAILED
>>>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>>>>>
>>>>>>>>> Total time: 17 seconds
>>>>>>>>> make[3]: *** [check-local] Error 1
>>>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>> make[2]: *** [check-am] Error 2
>>>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>> make[1]: *** [check-recursive] Error 1
>>>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>>>>>> make: *** [check-recursive] Error 1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> κρισhναν
>>>>>>>>>
>>>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>>>>>> c.bannister@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I've used the latest thrift to generate go code, and then
>>>>>>>>>> manually created executor config which works and is able to launch jobs.
>>>>>>>>>>
>>>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Krish
>>>>>>>>>>> We are using Thrift with go for all our api calls to Aurora,
>>>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>>>>>
>>>>>>>>>>> happy to help answer any questions you might have
>>>>>>>>>>>
>>>>>>>>>>> -Jake
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <
>>>>>>>>>>> krishnan.k.iyer@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks, Bill.
>>>>>>>>>>>>
>>>>>>>>>>>> Well I have started my foray into the the thrift API today. And
>>>>>>>>>>>> I think I am stuck with some thrift configs.
>>>>>>>>>>>>
>>>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to
>>>>>>>>>>>> talk with aurora using thrift 0.9.1? Are they compatible? I couldn't find
>>>>>>>>>>>> any changelog or compatibility statement on the thrift project site.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian
>>>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>>>>>>> krish@krish:/tmp
>>>>>>>>>>>> > thrift --gen go api.thrift
>>>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>>>>>
>>>>>>>>>>>> I can modify the golang code by hand, but I would like to play
>>>>>>>>>>>> it safe and use the working compiler from the debian repos.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into
>>>>>>>>>>>> a test golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>>>>>>>> similar error and gone past it?
>>>>>>>>>>>> I have already done a `go get
>>>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <
>>>>>>>>>>>> wfarner@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't
>>>>>>>>>>>>> much in the way of independent/holistic docs for the thrift API. There is,
>>>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>>>>>>> to that point!
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>>>>>> maxim@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Correct. There is also 'details' field that may have
>>>>>>>>>>>>>> additional messages (of error or informational nature)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort to
>>>>>>>>>>>>>> bring up a fully supported REST API that will presumably get documented and
>>>>>>>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ignore this object as it's an echo of the old client updater.
>>>>>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>>>>>> second life. Check out its new signature and comments in the api.thrift.
>>>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the existing
>>>>>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is to set role-level quota. Currently only required for
>>>>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more details.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>>>>> cluster.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>>>>>> instances of the job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Your observations are correct. This is the main API to change
>>>>>>>>>>>>>> your service job in any way (including adding, removing or modifying
>>>>>>>>>>>>>> instances).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can.
>>>>>>>>>>>>>> Those that will not find resources will remain in PENDING state until more
>>>>>>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>>>>>>> remain PENDING.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <
>>>>>>>>>>>>>> krishnan.k.iyer@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how
>>>>>>>>>>>>>>> to add new jobs.
>>>>>>>>>>>>>>> I am using aurora v0.12 released last month and have
>>>>>>>>>>>>>>> upgraded to mesos v0.25 accordingly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of
>>>>>>>>>>>>>>> some APIs, & would help it if someone can point out flaws in my
>>>>>>>>>>>>>>> understanding:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs
>>>>>>>>>>>>>>> specified, and return thrift values only in Response.result field.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to
>>>>>>>>>>>>>>> help understand Thrift API better?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>> This is basically the API to replace the Aurora
>>>>>>>>>>>>>>> DSL/.aurora files for job configuration.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>> By the naming convention, it seems this is used to
>>>>>>>>>>>>>>> increase the number of instances of a job. It will not result in stopping
>>>>>>>>>>>>>>> of current instances of the job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My second explanation for this API: Since it uses a set
>>>>>>>>>>>>>>> of instanceIds, this is used for adding already running job in slaves to
>>>>>>>>>>>>>>> the internal data structures of Aurora to track the job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>> Return the reason (in string) about why the job is
>>>>>>>>>>>>>>> PENDING. For example: insufficient CPU.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in
>>>>>>>>>>>>>>> the cluster.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string
>>>>>>>>>>>>>>> message):
>>>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig
>>>>>>>>>>>>>>> specified. Can be used if resource requirement changes. For example: If I
>>>>>>>>>>>>>>> wanted aurora to update the version of container used for a job using
>>>>>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Bill Farner <wf...@apache.org>.
Figured it out. I was first tipped off that the request being made is a
GET, while i've normally seen HTTP thrift clients making POST requests. I
haven't yet drilled into the rationale behind the alternative client in the
thrift library, but this fixed up your code on my trials:
- transport, err = thrift.NewTHttpClient(endpoint)
+ transport, err = thrift.NewTHttpPostClient(endpoint)
On Sun, Mar 20, 2016 at 10:58 PM, Krish <kr...@gmail.com> wrote:
> Thanks Bill.
> I checked the HTTP request. The sent packet data is:
> GET /api HTTP/1.1
> Host: 54.209.127.223:8081
> User-Agent: Go-http-client/1.1
> Accept-Encoding: gzip
>
> I can attach the complete capture if that is needed.
>
> Link to my code on github:
> https://github.com/krish7919/aurora_thrift_api
>
>
>
>
>
> --
> κρισhναν
>
> On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <wf...@apache.org> wrote:
>
>> Looks to me like it's failing before the request is even deserialized -
>> in the thrift layer (the exception is thrown here
>> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132>
>> - EOF). Can you capture the HTTP request that was sent by the client?
>> That might provide more clues.
>>
>> Also, if you're able to put up a git repo with your client code, i can
>> poke at it.
>>
>> On Sat, Mar 19, 2016 at 11:26 AM, Krish <kr...@gmail.com>
>> wrote:
>>
>>> Hi Bill,
>>> Tried digging more about aurora thrift API this weekend.
>>> The thrift generated code is a good reference point.
>>>
>>> You are right; the '/' path just gives me the list of URLs that one gets
>>> on the browser when I query it using my thrift client.
>>> Any pointers based on the data below will be very helpful to find out
>>> why is aurora bailing out when processing the thrift request, or is it some
>>> client side error that is causing it.
>>>
>>>
>>> My thrift client trying to query for getJobs:
>>>
>>> ....
>>> var protocolFactory thrift.TProtocolFactory
>>> var transport thrift.TTransport
>>> var client *api.ReadOnlySchedulerClient
>>> var err error
>>> transport, err = thrift.NewTHttpClient("
>>> http://54.209.17.221:8081/api")
>>> defer transport.Close()
>>> protocolFactory = thrift.NewTJSONProtocolFactory()
>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>> protocolFactory)
>>> err = transport.Open()
>>> if err != nil {
>>> fmt.Println("Error opening socket: ", err)
>>> os.Exit(1)
>>> }
>>> defer transport.Close()
>>> fmt.Println(client.GetJobs(""))
>>> ....
>>>
>>>
>>> I did a wireshark analysis of outgoing packets to aurora, & I do get a
>>> response packet from aurora, and my thrift client bails out with an error
>>> (runtime error: invalid memory address or nil pointer dereference). The
>>> data in the packet sent by server is:
>>>
>>> <html>
>>> <head>
>>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
>>> <title>Error 500 </title>
>>> </head>
>>> <body>
>>> <h2>HTTP ERROR: 500</h2>
>>> <p>Problem accessing /api. Reason:
>>> <pre> javax.servlet.ServletException:
>>> org.apache.thrift.transport.TTransportException</pre></p>
>>> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
>>> 9.3.6.v20151106</a><hr/>
>>> </body>
>>> </html>
>>>
>>>
>>>
>>>
>>> Stacktrace from the server/aurora console logs:
>>>
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
>>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
>>> org.apache.thrift.transport.TTransportException
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
>>> ~[aurora-0.12.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>> ~[aurora-0.12.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
>>> ~[aurora-0.12.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>> ~[aurora-0.12.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>> ~[guice-servlet-3.0.jar:na]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>> ...
>>> ...
>>> ...
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
>>> org.apache.thrift.transport.TTransportException: null
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>>> ~[libthrift-0.9.1.jar:0.9.1]
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted
>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
>>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
>>> [19/Mar/2016:18:12:23 +
>>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
>>>
>>>
>>>
>>>
>>> --
>>> κρισhναν
>>>
>>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <wf...@apache.org> wrote:
>>>
>>>> 8081 is indeed the default thrift port. If you can capture a raw HTTP
>>>> request, we could diagnose this better. My first hunch is that the thrift
>>>> client expects the API to be mounted at /, when in fact we mount it at /api.
>>>>
>>>> Jake can probably tell you exactly what's wrong based on his code, but
>>>> in the meantime you might find it helpful to compare against how we set up
>>>> the JS and python clients:
>>>>
>>>>
>>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>>>>
>>>>
>>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>>>>
>>>>
>>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Apologies for multiple mails, the previous email was sent accidentally.
>>>>> I didn't add a problem description, before hitting send.
>>>>>
>>>>> When I query aurora for all the jobs using getJobs, I find the aurora
>>>>> error as given below and the response I get using my client.
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Jake!
>>>>>>
>>>>>> That worked like a charm, & I was wondering why does install from
>>>>>> source doesn't work!
>>>>>>
>>>>>> Aurora Log:
>>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318
>>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>>>>
>>>>>> Thrift client:
>>>>>> ./mrfantastic_service.out
>>>>>> Connecting to aurora....
>>>>>> <nil> EOF
>>>>>>
>>>>>> Thrift client sourcec:
>>>>>> func thriftAuroraJobs() {
>>>>>> var protocolFactory thrift.TProtocolFactory
>>>>>> var transportFactory thrift.TTransportFactory
>>>>>> var transport thrift.TTransport
>>>>>> var client *api.ReadOnlySchedulerClient
>>>>>> var err error
>>>>>>
>>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>>>>> //transportFactory =
>>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>>>>> transportFactory = thrift.NewTTransportFactory()
>>>>>> fmt.Println("Connecting to aurora....")
>>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>>>>> if err != nil {
>>>>>> fmt.Println("Error opening socket:", err)
>>>>>> os.Exit(1)
>>>>>> //return err
>>>>>> }
>>>>>> if transport == nil {
>>>>>> os.Exit(1)
>>>>>> }
>>>>>> transport = transportFactory.GetTransport(transport)
>>>>>> if transport == nil {
>>>>>> os.Exit(1)
>>>>>> }
>>>>>> err = transport.Open()
>>>>>> if err != nil {
>>>>>> os.Exit(1)
>>>>>> }
>>>>>> defer transport.Close()
>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>>> protocolFactory)
>>>>>> fmt.Println(client.GetJobs(""))
>>>>>> }
>>>>>>
>>>>>>
>>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora is
>>>>>> looking for HTTP data on the socket.
>>>>>> Is there a config that needs to be set for thrift API to be
>>>>>> initialized?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> κρισhναν
>>>>>>
>>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> if you just need the compiler you can install the thrift-compiler
>>>>>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>>>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>>>>
>>>>>>> -Jake
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> deb packaging (tested with ubuntu trusty)
>>>>>>>
>>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>>>>> > /etc/apt/sources.list.d/thrift.list
>>>>>>>
>>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>>>>
>>>>>>>
>>>>>>> or for centos/rhel (tested with centos 7.2)
>>>>>>>
>>>>>>> > /etc/yum.repos.d/thrift.repo
>>>>>>>
>>>>>>> [thrift]
>>>>>>> name=Apache Thrift rpm repo
>>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>>>>> enabled=1
>>>>>>> gpgcheck=0
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Jake/Chris,
>>>>>>>> Thanks for the info.
>>>>>>>> When I try to install thrift v0.9.3 from source, I get an error as
>>>>>>>> follows while running `make check`:
>>>>>>>> ...
>>>>>>>> ...
>>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>>>>> elapsed: 0.062 sec
>>>>>>>> [junit] Running
>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>>>>>>> elapsed: 0.046 sec
>>>>>>>>
>>>>>>>> BUILD FAILED
>>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>>>>
>>>>>>>> Total time: 17 seconds
>>>>>>>> make[3]: *** [check-local] Error 1
>>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>> make[2]: *** [check-am] Error 2
>>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>> make[1]: *** [check-recursive] Error 1
>>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>>>>> make: *** [check-recursive] Error 1
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> κρισhναν
>>>>>>>>
>>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>>>>> c.bannister@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I've used the latest thrift to generate go code, and then manually
>>>>>>>>> created executor config which works and is able to launch jobs.
>>>>>>>>>
>>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Krish
>>>>>>>>>> We are using Thrift with go for all our api calls to Aurora,
>>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>>>>
>>>>>>>>>> happy to help answer any questions you might have
>>>>>>>>>>
>>>>>>>>>> -Jake
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <krishnan.k.iyer@gmail.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks, Bill.
>>>>>>>>>>>
>>>>>>>>>>> Well I have started my foray into the the thrift API today. And
>>>>>>>>>>> I think I am stuck with some thrift configs.
>>>>>>>>>>>
>>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk
>>>>>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>>>>>>>> changelog or compatibility statement on the thrift project site.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian
>>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>>>>>> krish@krish:/tmp
>>>>>>>>>>> > thrift --gen go api.thrift
>>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>>>>
>>>>>>>>>>> I can modify the golang code by hand, but I would like to play
>>>>>>>>>>> it safe and use the working compiler from the debian repos.
>>>>>>>>>>>
>>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a
>>>>>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>>>>>>> similar error and gone past it?
>>>>>>>>>>> I have already done a `go get
>>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> κρισhναν
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <
>>>>>>>>>>> wfarner@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't
>>>>>>>>>>>> much in the way of independent/holistic docs for the thrift API. There is,
>>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>>>>
>>>>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>>>>>> to that point!
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>>>>> maxim@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Correct. There is also 'details' field that may have
>>>>>>>>>>>>> additional messages (of error or informational nature)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort to
>>>>>>>>>>>>> bring up a fully supported REST API that will presumably get documented and
>>>>>>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ignore this object as it's an echo of the old client updater.
>>>>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>>>>> second life. Check out its new signature and comments in the api.thrift.
>>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the existing
>>>>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is to set role-level quota. Currently only required for
>>>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more details.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>>>> cluster.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>>>>> instances of the job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your observations are correct. This is the main API to change
>>>>>>>>>>>>> your service job in any way (including adding, removing or modifying
>>>>>>>>>>>>> instances).
>>>>>>>>>>>>>
>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can.
>>>>>>>>>>>>> Those that will not find resources will remain in PENDING state until more
>>>>>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>>>>>> remain PENDING.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <
>>>>>>>>>>>>> krishnan.k.iyer@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how to
>>>>>>>>>>>>>> add new jobs.
>>>>>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded
>>>>>>>>>>>>>> to mesos v0.25 accordingly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of some
>>>>>>>>>>>>>> APIs, & would help it if someone can point out flaws in my understanding:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs
>>>>>>>>>>>>>> specified, and return thrift values only in Response.result field.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to
>>>>>>>>>>>>>> help understand Thrift API better?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>> This is basically the API to replace the Aurora
>>>>>>>>>>>>>> DSL/.aurora files for job configuration.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>> By the naming convention, it seems this is used to
>>>>>>>>>>>>>> increase the number of instances of a job. It will not result in stopping
>>>>>>>>>>>>>> of current instances of the job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My second explanation for this API: Since it uses a set
>>>>>>>>>>>>>> of instanceIds, this is used for adding already running job in slaves to
>>>>>>>>>>>>>> the internal data structures of Aurora to track the job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>> Return the reason (in string) about why the job is
>>>>>>>>>>>>>> PENDING. For example: insufficient CPU.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in
>>>>>>>>>>>>>> the cluster.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string
>>>>>>>>>>>>>> message):
>>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig specified.
>>>>>>>>>>>>>> Can be used if resource requirement changes. For example: If I wanted
>>>>>>>>>>>>>> aurora to update the version of container used for a job using
>>>>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Thanks Bill.
I checked the HTTP request. The sent packet data is:
GET /api HTTP/1.1
Host: 54.209.127.223:8081
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip
I can attach the complete capture if that is needed.
Link to my code on github:
https://github.com/krish7919/aurora_thrift_api
--
κρισhναν
On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <wf...@apache.org> wrote:
> Looks to me like it's failing before the request is even deserialized - in
> the thrift layer (the exception is thrown here
> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132>
> - EOF). Can you capture the HTTP request that was sent by the client?
> That might provide more clues.
>
> Also, if you're able to put up a git repo with your client code, i can
> poke at it.
>
> On Sat, Mar 19, 2016 at 11:26 AM, Krish <kr...@gmail.com> wrote:
>
>> Hi Bill,
>> Tried digging more about aurora thrift API this weekend.
>> The thrift generated code is a good reference point.
>>
>> You are right; the '/' path just gives me the list of URLs that one gets
>> on the browser when I query it using my thrift client.
>> Any pointers based on the data below will be very helpful to find out why
>> is aurora bailing out when processing the thrift request, or is it some
>> client side error that is causing it.
>>
>>
>> My thrift client trying to query for getJobs:
>>
>> ....
>> var protocolFactory thrift.TProtocolFactory
>> var transport thrift.TTransport
>> var client *api.ReadOnlySchedulerClient
>> var err error
>> transport, err = thrift.NewTHttpClient("http://54.209.17.221:8081/api
>> ")
>> defer transport.Close()
>> protocolFactory = thrift.NewTJSONProtocolFactory()
>> client = api.NewReadOnlySchedulerClientFactory(transport,
>> protocolFactory)
>> err = transport.Open()
>> if err != nil {
>> fmt.Println("Error opening socket: ", err)
>> os.Exit(1)
>> }
>> defer transport.Close()
>> fmt.Println(client.GetJobs(""))
>> ....
>>
>>
>> I did a wireshark analysis of outgoing packets to aurora, & I do get a
>> response packet from aurora, and my thrift client bails out with an error
>> (runtime error: invalid memory address or nil pointer dereference). The
>> data in the packet sent by server is:
>>
>> <html>
>> <head>
>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
>> <title>Error 500 </title>
>> </head>
>> <body>
>> <h2>HTTP ERROR: 500</h2>
>> <p>Problem accessing /api. Reason:
>> <pre> javax.servlet.ServletException:
>> org.apache.thrift.transport.TTransportException</pre></p>
>> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
>> 9.3.6.v20151106</a><hr/>
>> </body>
>> </html>
>>
>>
>>
>>
>> Stacktrace from the server/aurora console logs:
>>
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
>> org.apache.thrift.transport.TTransportException
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
>> ~[aurora-0.12.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>> ~[aurora-0.12.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
>> ~[aurora-0.12.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>> ~[aurora-0.12.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>> ~[guice-servlet-3.0.jar:na]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>> ...
>> ...
>> ...
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
>> org.apache.thrift.transport.TTransportException: null
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>> ~[libthrift-0.9.1.jar:0.9.1]
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted
>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
>> [19/Mar/2016:18:12:23 +
>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <wf...@apache.org> wrote:
>>
>>> 8081 is indeed the default thrift port. If you can capture a raw HTTP
>>> request, we could diagnose this better. My first hunch is that the thrift
>>> client expects the API to be mounted at /, when in fact we mount it at /api.
>>>
>>> Jake can probably tell you exactly what's wrong based on his code, but
>>> in the meantime you might find it helpful to compare against how we set up
>>> the JS and python clients:
>>>
>>>
>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>>>
>>>
>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>>>
>>>
>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Apologies for multiple mails, the previous email was sent accidentally.
>>>> I didn't add a problem description, before hitting send.
>>>>
>>>> When I query aurora for all the jobs using getJobs, I find the aurora
>>>> error as given below and the response I get using my client.
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Jake!
>>>>>
>>>>> That worked like a charm, & I was wondering why does install from
>>>>> source doesn't work!
>>>>>
>>>>> Aurora Log:
>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318
>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>>>
>>>>> Thrift client:
>>>>> ./mrfantastic_service.out
>>>>> Connecting to aurora....
>>>>> <nil> EOF
>>>>>
>>>>> Thrift client sourcec:
>>>>> func thriftAuroraJobs() {
>>>>> var protocolFactory thrift.TProtocolFactory
>>>>> var transportFactory thrift.TTransportFactory
>>>>> var transport thrift.TTransport
>>>>> var client *api.ReadOnlySchedulerClient
>>>>> var err error
>>>>>
>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>>>> //transportFactory =
>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>>>> transportFactory = thrift.NewTTransportFactory()
>>>>> fmt.Println("Connecting to aurora....")
>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>>>> if err != nil {
>>>>> fmt.Println("Error opening socket:", err)
>>>>> os.Exit(1)
>>>>> //return err
>>>>> }
>>>>> if transport == nil {
>>>>> os.Exit(1)
>>>>> }
>>>>> transport = transportFactory.GetTransport(transport)
>>>>> if transport == nil {
>>>>> os.Exit(1)
>>>>> }
>>>>> err = transport.Open()
>>>>> if err != nil {
>>>>> os.Exit(1)
>>>>> }
>>>>> defer transport.Close()
>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>> protocolFactory)
>>>>> fmt.Println(client.GetJobs(""))
>>>>> }
>>>>>
>>>>>
>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora is
>>>>> looking for HTTP data on the socket.
>>>>> Is there a config that needs to be set for thrift API to be
>>>>> initialized?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> if you just need the compiler you can install the thrift-compiler
>>>>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>>>
>>>>>> -Jake
>>>>>>
>>>>>>
>>>>>>
>>>>>> deb packaging (tested with ubuntu trusty)
>>>>>>
>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>>>> > /etc/apt/sources.list.d/thrift.list
>>>>>>
>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>>>
>>>>>>
>>>>>> or for centos/rhel (tested with centos 7.2)
>>>>>>
>>>>>> > /etc/yum.repos.d/thrift.repo
>>>>>>
>>>>>> [thrift]
>>>>>> name=Apache Thrift rpm repo
>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>>>> enabled=1
>>>>>> gpgcheck=0
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Jake/Chris,
>>>>>>> Thanks for the info.
>>>>>>> When I try to install thrift v0.9.3 from source, I get an error as
>>>>>>> follows while running `make check`:
>>>>>>> ...
>>>>>>> ...
>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>>>> elapsed: 0.062 sec
>>>>>>> [junit] Running
>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>>>>>> elapsed: 0.046 sec
>>>>>>>
>>>>>>> BUILD FAILED
>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>>>
>>>>>>> Total time: 17 seconds
>>>>>>> make[3]: *** [check-local] Error 1
>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>> make[2]: *** [check-am] Error 2
>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>> make[1]: *** [check-recursive] Error 1
>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>>>> make: *** [check-recursive] Error 1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>>>> c.bannister@gmail.com> wrote:
>>>>>>>
>>>>>>>> I've used the latest thrift to generate go code, and then manually
>>>>>>>> created executor config which works and is able to launch jobs.
>>>>>>>>
>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Krish
>>>>>>>>> We are using Thrift with go for all our api calls to Aurora, would
>>>>>>>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>>>
>>>>>>>>> happy to help answer any questions you might have
>>>>>>>>>
>>>>>>>>> -Jake
>>>>>>>>>
>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, Bill.
>>>>>>>>>>
>>>>>>>>>> Well I have started my foray into the the thrift API today. And I
>>>>>>>>>> think I am stuck with some thrift configs.
>>>>>>>>>>
>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk
>>>>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>>>>>>> changelog or compatibility statement on the thrift project site.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian
>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>>>>> krish@krish:/tmp
>>>>>>>>>> > thrift --gen go api.thrift
>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>>>
>>>>>>>>>> I can modify the golang code by hand, but I would like to play it
>>>>>>>>>> safe and use the working compiler from the debian repos.
>>>>>>>>>>
>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a
>>>>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>>>>>> similar error and gone past it?
>>>>>>>>>> I have already done a `go get
>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> κρισhναν
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wfarner@apache.org
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't much
>>>>>>>>>>> in the way of independent/holistic docs for the thrift API. There is,
>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>>>
>>>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>>>>> to that point!
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>>>> maxim@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>>>>>>> return thrift values only in Response.result field.
>>>>>>>>>>>>
>>>>>>>>>>>> Correct. There is also 'details' field that may have additional
>>>>>>>>>>>> messages (of error or informational nature)
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>
>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort to
>>>>>>>>>>>> bring up a fully supported REST API that will presumably get documented and
>>>>>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>
>>>>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>
>>>>>>>>>>>> Ignore this object as it's an echo of the old client updater.
>>>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>>>>>>
>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>
>>>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>>>> second life. Check out its new signature and comments in the api.thrift.
>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the existing
>>>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>>>
>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>
>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>>>
>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>
>>>>>>>>>>>> This is to set role-level quota. Currently only required for
>>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more details.
>>>>>>>>>>>>
>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>>> cluster.
>>>>>>>>>>>>
>>>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>>>> instances of the job.
>>>>>>>>>>>>
>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>>>
>>>>>>>>>>>> Your observations are correct. This is the main API to change
>>>>>>>>>>>> your service job in any way (including adding, removing or modifying
>>>>>>>>>>>> instances).
>>>>>>>>>>>>
>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>
>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can.
>>>>>>>>>>>> Those that will not find resources will remain in PENDING state until more
>>>>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>>>>> remain PENDING.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <
>>>>>>>>>>>> krishnan.k.iyer@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how to
>>>>>>>>>>>>> add new jobs.
>>>>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded
>>>>>>>>>>>>> to mesos v0.25 accordingly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of some
>>>>>>>>>>>>> APIs, & would help it if someone can point out flaws in my understanding:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs
>>>>>>>>>>>>> specified, and return thrift values only in Response.result field.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>> This is basically the API to replace the Aurora
>>>>>>>>>>>>> DSL/.aurora files for job configuration.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>> By the naming convention, it seems this is used to
>>>>>>>>>>>>> increase the number of instances of a job. It will not result in stopping
>>>>>>>>>>>>> of current instances of the job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>> Return the reason (in string) about why the job is
>>>>>>>>>>>>> PENDING. For example: insufficient CPU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>>> cluster.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string
>>>>>>>>>>>>> message):
>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig specified.
>>>>>>>>>>>>> Can be used if resource requirement changes. For example: If I wanted
>>>>>>>>>>>>> aurora to update the version of container used for a job using
>>>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Bill Farner <wf...@apache.org>.
Looks to me like it's failing before the request is even deserialized - in
the thrift layer (the exception is thrown here
<https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132>
- EOF). Can you capture the HTTP request that was sent by the client?
That might provide more clues.
Also, if you're able to put up a git repo with your client code, i can poke
at it.
On Sat, Mar 19, 2016 at 11:26 AM, Krish <kr...@gmail.com> wrote:
> Hi Bill,
> Tried digging more about aurora thrift API this weekend.
> The thrift generated code is a good reference point.
>
> You are right; the '/' path just gives me the list of URLs that one gets
> on the browser when I query it using my thrift client.
> Any pointers based on the data below will be very helpful to find out why
> is aurora bailing out when processing the thrift request, or is it some
> client side error that is causing it.
>
>
> My thrift client trying to query for getJobs:
>
> ....
> var protocolFactory thrift.TProtocolFactory
> var transport thrift.TTransport
> var client *api.ReadOnlySchedulerClient
> var err error
> transport, err = thrift.NewTHttpClient("http://54.209.17.221:8081/api
> ")
> defer transport.Close()
> protocolFactory = thrift.NewTJSONProtocolFactory()
> client = api.NewReadOnlySchedulerClientFactory(transport,
> protocolFactory)
> err = transport.Open()
> if err != nil {
> fmt.Println("Error opening socket: ", err)
> os.Exit(1)
> }
> defer transport.Close()
> fmt.Println(client.GetJobs(""))
> ....
>
>
> I did a wireshark analysis of outgoing packets to aurora, & I do get a
> response packet from aurora, and my thrift client bails out with an error
> (runtime error: invalid memory address or nil pointer dereference). The
> data in the packet sent by server is:
>
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> <title>Error 500 </title>
> </head>
> <body>
> <h2>HTTP ERROR: 500</h2>
> <p>Problem accessing /api. Reason:
> <pre> javax.servlet.ServletException:
> org.apache.thrift.transport.TTransportException</pre></p>
> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
> 9.3.6.v20151106</a><hr/>
> </body>
> </html>
>
>
>
>
> Stacktrace from the server/aurora console logs:
>
> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
> org.apache.thrift.transport.TTransportException
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> ~[javax.servlet-api-3.1.0.jar:3.1.0]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> ~[javax.servlet-api-3.1.0.jar:3.1.0]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
> ~[aurora-0.12.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
> ~[aurora-0.12.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
> ~[aurora-0.12.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
> ~[aurora-0.12.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> ~[guice-servlet-3.0.jar:na]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
> ...
> ...
> ...
> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
> org.apache.thrift.transport.TTransportException: null
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
> org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> ~[libthrift-0.9.1.jar:0.9.1]
> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted
> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
> [19/Mar/2016:18:12:23 +
> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
>
>
>
>
> --
> κρισhναν
>
> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <wf...@apache.org> wrote:
>
>> 8081 is indeed the default thrift port. If you can capture a raw HTTP
>> request, we could diagnose this better. My first hunch is that the thrift
>> client expects the API to be mounted at /, when in fact we mount it at /api.
>>
>> Jake can probably tell you exactly what's wrong based on his code, but in
>> the meantime you might find it helpful to compare against how we set up the
>> JS and python clients:
>>
>>
>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>>
>>
>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>>
>>
>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <kr...@gmail.com> wrote:
>>
>>> Apologies for multiple mails, the previous email was sent accidentally.
>>> I didn't add a problem description, before hitting send.
>>>
>>> When I query aurora for all the jobs using getJobs, I find the aurora
>>> error as given below and the response I get using my client.
>>>
>>> --
>>> κρισhναν
>>>
>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Jake!
>>>>
>>>> That worked like a charm, & I was wondering why does install from
>>>> source doesn't work!
>>>>
>>>> Aurora Log:
>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109
>>>> [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>>
>>>> Thrift client:
>>>> ./mrfantastic_service.out
>>>> Connecting to aurora....
>>>> <nil> EOF
>>>>
>>>> Thrift client sourcec:
>>>> func thriftAuroraJobs() {
>>>> var protocolFactory thrift.TProtocolFactory
>>>> var transportFactory thrift.TTransportFactory
>>>> var transport thrift.TTransport
>>>> var client *api.ReadOnlySchedulerClient
>>>> var err error
>>>>
>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>>> //transportFactory =
>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>>> transportFactory = thrift.NewTTransportFactory()
>>>> fmt.Println("Connecting to aurora....")
>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>>> if err != nil {
>>>> fmt.Println("Error opening socket:", err)
>>>> os.Exit(1)
>>>> //return err
>>>> }
>>>> if transport == nil {
>>>> os.Exit(1)
>>>> }
>>>> transport = transportFactory.GetTransport(transport)
>>>> if transport == nil {
>>>> os.Exit(1)
>>>> }
>>>> err = transport.Open()
>>>> if err != nil {
>>>> os.Exit(1)
>>>> }
>>>> defer transport.Close()
>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>> protocolFactory)
>>>> fmt.Println(client.GetJobs(""))
>>>> }
>>>>
>>>>
>>>> My hunch is that the thrift port isn't 8081, as the server/aurora is
>>>> looking for HTTP data on the socket.
>>>> Is there a config that needs to be set for thrift API to be initialized?
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org>
>>>> wrote:
>>>>
>>>>> if you just need the compiler you can install the thrift-compiler
>>>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>>
>>>>> -Jake
>>>>>
>>>>>
>>>>>
>>>>> deb packaging (tested with ubuntu trusty)
>>>>>
>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>>> > /etc/apt/sources.list.d/thrift.list
>>>>>
>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>>
>>>>>
>>>>> or for centos/rhel (tested with centos 7.2)
>>>>>
>>>>> > /etc/yum.repos.d/thrift.repo
>>>>>
>>>>> [thrift]
>>>>> name=Apache Thrift rpm repo
>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>>> enabled=1
>>>>> gpgcheck=0
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Jake/Chris,
>>>>>> Thanks for the info.
>>>>>> When I try to install thrift v0.9.3 from source, I get an error as
>>>>>> follows while running `make check`:
>>>>>> ...
>>>>>> ...
>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>>> elapsed: 0.062 sec
>>>>>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>>>>> elapsed: 0.046 sec
>>>>>>
>>>>>> BUILD FAILED
>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>>
>>>>>> Total time: 17 seconds
>>>>>> make[3]: *** [check-local] Error 1
>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>> make[2]: *** [check-am] Error 2
>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>> make[1]: *** [check-recursive] Error 1
>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>>> make: *** [check-recursive] Error 1
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> κρισhναν
>>>>>>
>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>>> c.bannister@gmail.com> wrote:
>>>>>>
>>>>>>> I've used the latest thrift to generate go code, and then manually
>>>>>>> created executor config which works and is able to launch jobs.
>>>>>>>
>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Krish
>>>>>>>> We are using Thrift with go for all our api calls to Aurora, would
>>>>>>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>>
>>>>>>>> happy to help answer any questions you might have
>>>>>>>>
>>>>>>>> -Jake
>>>>>>>>
>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks, Bill.
>>>>>>>>>
>>>>>>>>> Well I have started my foray into the the thrift API today. And I
>>>>>>>>> think I am stuck with some thrift configs.
>>>>>>>>>
>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk
>>>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>>>>>> changelog or compatibility statement on the thrift project site.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos
>>>>>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>>>> krish@krish:/tmp
>>>>>>>>> > thrift --gen go api.thrift
>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>>
>>>>>>>>> I can modify the golang code by hand, but I would like to play it
>>>>>>>>> safe and use the working compiler from the debian repos.
>>>>>>>>>
>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a
>>>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>>>>> similar error and gone past it?
>>>>>>>>> I have already done a `go get
>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> κρισhναν
>>>>>>>>>
>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't much
>>>>>>>>>> in the way of independent/holistic docs for the thrift API. There is,
>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>>
>>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>>>> to that point!
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>>> maxim@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>>>>>> return thrift values only in Response.result field.
>>>>>>>>>>>
>>>>>>>>>>> Correct. There is also 'details' field that may have additional
>>>>>>>>>>> messages (of error or informational nature)
>>>>>>>>>>>
>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>
>>>>>>>>>>> The thrift API is largely undocumented. There is an effort to
>>>>>>>>>>> bring up a fully supported REST API that will presumably get documented and
>>>>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>>>>
>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>
>>>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>>
>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>>>>>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>>>>>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>>>>>>>>> with the same JobKey.
>>>>>>>>>>>
>>>>>>>>>>> Ignore this object as it's an echo of the old client updater.
>>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>>>>>
>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>
>>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>>> second life. Check out its new signature and comments in the api.thrift.
>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the existing
>>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>>
>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>
>>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>>
>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>
>>>>>>>>>>> This is to set role-level quota. Currently only required for
>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more details.
>>>>>>>>>>>
>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>> cluster.
>>>>>>>>>>>
>>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>>> instances of the job.
>>>>>>>>>>>
>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>>
>>>>>>>>>>> Your observations are correct. This is the main API to change
>>>>>>>>>>> your service job in any way (including adding, removing or modifying
>>>>>>>>>>> instances).
>>>>>>>>>>>
>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>
>>>>>>>>>>> Scheduler will try to schedule as many instances as it can.
>>>>>>>>>>> Those that will not find resources will remain in PENDING state until more
>>>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>>>> remain PENDING.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <
>>>>>>>>>>> krishnan.k.iyer@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how to
>>>>>>>>>>>> add new jobs.
>>>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded
>>>>>>>>>>>> to mesos v0.25 accordingly.
>>>>>>>>>>>>
>>>>>>>>>>>> Below is a summary of my (very limited) understanding of some
>>>>>>>>>>>> APIs, & would help it if someone can point out flaws in my understanding:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>
>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>>>>>>> files for job configuration.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>
>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>> By the naming convention, it seems this is used to increase
>>>>>>>>>>>> the number of instances of a job. It will not result in stopping of current
>>>>>>>>>>>> instances of the job.
>>>>>>>>>>>>
>>>>>>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>>>>>>
>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>> Return the reason (in string) about why the job is PENDING.
>>>>>>>>>>>> For example: insufficient CPU.
>>>>>>>>>>>>
>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>
>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>> cluster.
>>>>>>>>>>>>
>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>>> Used for updating jobs with the new TaskConfig specified.
>>>>>>>>>>>> Can be used if resource requirement changes. For example: If I wanted
>>>>>>>>>>>> aurora to update the version of container used for a job using
>>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Hi Bill,
Tried digging more about aurora thrift API this weekend.
The thrift generated code is a good reference point.
You are right; the '/' path just gives me the list of URLs that one gets on
the browser when I query it using my thrift client.
Any pointers based on the data below will be very helpful to find out why
is aurora bailing out when processing the thrift request, or is it some
client side error that is causing it.
My thrift client trying to query for getJobs:
....
var protocolFactory thrift.TProtocolFactory
var transport thrift.TTransport
var client *api.ReadOnlySchedulerClient
var err error
transport, err = thrift.NewTHttpClient("http://54.209.17.221:8081/api")
defer transport.Close()
protocolFactory = thrift.NewTJSONProtocolFactory()
client = api.NewReadOnlySchedulerClientFactory(transport,
protocolFactory)
err = transport.Open()
if err != nil {
fmt.Println("Error opening socket: ", err)
os.Exit(1)
}
defer transport.Close()
fmt.Println(client.GetJobs(""))
....
I did a wireshark analysis of outgoing packets to aurora, & I do get a
response packet from aurora, and my thrift client bails out with an error
(runtime error: invalid memory address or nil pointer dereference). The
data in the packet sent by server is:
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /api. Reason:
<pre> javax.servlet.ServletException:
org.apache.thrift.transport.TTransportException</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
9.3.6.v20151106</a><hr/>
</body>
</html>
Stacktrace from the server/aurora console logs:
Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
[qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
org.apache.thrift.transport.TTransportException
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
~[javax.servlet-api-3.1.0.jar:3.1.0]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
~[javax.servlet-api-3.1.0.jar:3.1.0]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
~[aurora-0.12.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
~[aurora-0.12.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
~[aurora-0.12.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
~[aurora-0.12.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
~[guice-servlet-3.0.jar:na]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
[jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
[jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
...
...
...
Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
org.apache.thrift.transport.TTransportException: null
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: at
org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
~[libthrift-0.9.1.jar:0.9.1]
Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted
Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
[qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
[19/Mar/2016:18:12:23 +
0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
--
κρισhναν
On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <wf...@apache.org> wrote:
> 8081 is indeed the default thrift port. If you can capture a raw HTTP
> request, we could diagnose this better. My first hunch is that the thrift
> client expects the API to be mounted at /, when in fact we mount it at /api.
>
> Jake can probably tell you exactly what's wrong based on his code, but in
> the meantime you might find it helpful to compare against how we set up the
> JS and python clients:
>
>
> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>
>
> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>
>
> On Fri, Mar 18, 2016 at 7:50 AM, Krish <kr...@gmail.com> wrote:
>
>> Apologies for multiple mails, the previous email was sent accidentally.
>> I didn't add a problem description, before hitting send.
>>
>> When I query aurora for all the jobs using getJobs, I find the aurora
>> error as given below and the response I get using my client.
>>
>> --
>> κρισhναν
>>
>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com> wrote:
>>
>>> Thanks Jake!
>>>
>>> That worked like a charm, & I was wondering why does install from source
>>> doesn't work!
>>>
>>> Aurora Log:
>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109
>>> [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>
>>> Thrift client:
>>> ./mrfantastic_service.out
>>> Connecting to aurora....
>>> <nil> EOF
>>>
>>> Thrift client sourcec:
>>> func thriftAuroraJobs() {
>>> var protocolFactory thrift.TProtocolFactory
>>> var transportFactory thrift.TTransportFactory
>>> var transport thrift.TTransport
>>> var client *api.ReadOnlySchedulerClient
>>> var err error
>>>
>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>> //transportFactory =
>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>> transportFactory = thrift.NewTTransportFactory()
>>> fmt.Println("Connecting to aurora....")
>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>> if err != nil {
>>> fmt.Println("Error opening socket:", err)
>>> os.Exit(1)
>>> //return err
>>> }
>>> if transport == nil {
>>> os.Exit(1)
>>> }
>>> transport = transportFactory.GetTransport(transport)
>>> if transport == nil {
>>> os.Exit(1)
>>> }
>>> err = transport.Open()
>>> if err != nil {
>>> os.Exit(1)
>>> }
>>> defer transport.Close()
>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>> protocolFactory)
>>> fmt.Println(client.GetJobs(""))
>>> }
>>>
>>>
>>> My hunch is that the thrift port isn't 8081, as the server/aurora is
>>> looking for HTTP data on the socket.
>>> Is there a config that needs to be set for thrift API to be initialized?
>>>
>>>
>>>
>>>
>>> --
>>> κρισhναν
>>>
>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org>
>>> wrote:
>>>
>>>> if you just need the compiler you can install the thrift-compiler
>>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>
>>>> -Jake
>>>>
>>>>
>>>>
>>>> deb packaging (tested with ubuntu trusty)
>>>>
>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>> > /etc/apt/sources.list.d/thrift.list
>>>>
>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>
>>>>
>>>> or for centos/rhel (tested with centos 7.2)
>>>>
>>>> > /etc/yum.repos.d/thrift.repo
>>>>
>>>> [thrift]
>>>> name=Apache Thrift rpm repo
>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>> enabled=1
>>>> gpgcheck=0
>>>>
>>>>
>>>>
>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Jake/Chris,
>>>>> Thanks for the info.
>>>>> When I try to install thrift v0.9.3 from source, I get an error as
>>>>> follows while running `make check`:
>>>>> ...
>>>>> ...
>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>> elapsed: 0.062 sec
>>>>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>>>> elapsed: 0.046 sec
>>>>>
>>>>> BUILD FAILED
>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>
>>>>> Total time: 17 seconds
>>>>> make[3]: *** [check-local] Error 1
>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>> make[2]: *** [check-am] Error 2
>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>> make[1]: *** [check-recursive] Error 1
>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>> make: *** [check-recursive] Error 1
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>> c.bannister@gmail.com> wrote:
>>>>>
>>>>>> I've used the latest thrift to generate go code, and then manually
>>>>>> created executor config which works and is able to launch jobs.
>>>>>>
>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Krish
>>>>>>> We are using Thrift with go for all our api calls to Aurora, would
>>>>>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>
>>>>>>> happy to help answer any questions you might have
>>>>>>>
>>>>>>> -Jake
>>>>>>>
>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks, Bill.
>>>>>>>>
>>>>>>>> Well I have started my foray into the the thrift API today. And I
>>>>>>>> think I am stuck with some thrift configs.
>>>>>>>>
>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk
>>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>>>>> changelog or compatibility statement on the thrift project site.
>>>>>>>>
>>>>>>>>
>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos
>>>>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>>> krish@krish:/tmp
>>>>>>>> > thrift --gen go api.thrift
>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>
>>>>>>>> I can modify the golang code by hand, but I would like to play it
>>>>>>>> safe and use the working compiler from the debian repos.
>>>>>>>>
>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a
>>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>>>> similar error and gone past it?
>>>>>>>> I have already done a `go get
>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> κρισhναν
>>>>>>>>
>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Regarding documentation - Maxim is correct that there isn't much
>>>>>>>>> in the way of independent/holistic docs for the thrift API. There is,
>>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>
>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>>> to that point!
>>>>>>>>>
>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>> maxim@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>>>>> return thrift values only in Response.result field.
>>>>>>>>>>
>>>>>>>>>> Correct. There is also 'details' field that may have additional
>>>>>>>>>> messages (of error or informational nature)
>>>>>>>>>>
>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>
>>>>>>>>>> The thrift API is largely undocumented. There is an effort to
>>>>>>>>>> bring up a fully supported REST API that will presumably get documented and
>>>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>>>
>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>
>>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>
>>>>>>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>>>>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>>>>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>>>>>>>> with the same JobKey.
>>>>>>>>>>
>>>>>>>>>> Ignore this object as it's an echo of the old client updater.
>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>>>>
>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>
>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>> second life. Check out its new signature and comments in the api.thrift.
>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the existing
>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>
>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>
>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>
>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>
>>>>>>>>>> This is to set role-level quota. Currently only required for
>>>>>>>>>> tasks with 'production=True'. Search through our docs for more details.
>>>>>>>>>>
>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>> cluster.
>>>>>>>>>>
>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>> instances of the job.
>>>>>>>>>>
>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>
>>>>>>>>>> Your observations are correct. This is the main API to change
>>>>>>>>>> your service job in any way (including adding, removing or modifying
>>>>>>>>>> instances).
>>>>>>>>>>
>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>
>>>>>>>>>> Scheduler will try to schedule as many instances as it can. Those
>>>>>>>>>> that will not find resources will remain in PENDING state until more
>>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>>> remain PENDING.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <krishnan.k.iyer@gmail.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> I was going through the Aurora Thrift API to determine how to
>>>>>>>>>>> add new jobs.
>>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>>>>>>> mesos v0.25 accordingly.
>>>>>>>>>>>
>>>>>>>>>>> Below is a summary of my (very limited) understanding of some
>>>>>>>>>>> APIs, & would help it if someone can point out flaws in my understanding:
>>>>>>>>>>>
>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>>
>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>
>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>>>>>> files for job configuration.
>>>>>>>>>>>
>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>
>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>> By the naming convention, it seems this is used to increase
>>>>>>>>>>> the number of instances of a job. It will not result in stopping of current
>>>>>>>>>>> instances of the job.
>>>>>>>>>>>
>>>>>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>>>>>
>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>> Return the reason (in string) about why the job is PENDING.
>>>>>>>>>>> For example: insufficient CPU.
>>>>>>>>>>>
>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>
>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>> cluster.
>>>>>>>>>>>
>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>> Used for updating jobs with the new TaskConfig specified.
>>>>>>>>>>> Can be used if resource requirement changes. For example: If I wanted
>>>>>>>>>>> aurora to update the version of container used for a job using
>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> κρισhναν
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Bill Farner <wf...@apache.org>.
8081 is indeed the default thrift port. If you can capture a raw HTTP
request, we could diagnose this better. My first hunch is that the thrift
client expects the API to be mounted at /, when in fact we mount it at /api.
Jake can probably tell you exactly what's wrong based on his code, but in
the meantime you might find it helpful to compare against how we set up the
JS and python clients:
https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
On Fri, Mar 18, 2016 at 7:50 AM, Krish <kr...@gmail.com> wrote:
> Apologies for multiple mails, the previous email was sent accidentally.
> I didn't add a problem description, before hitting send.
>
> When I query aurora for all the jobs using getJobs, I find the aurora
> error as given below and the response I get using my client.
>
> --
> κρισhναν
>
> On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com> wrote:
>
>> Thanks Jake!
>>
>> That worked like a charm, & I was wondering why does install from source
>> doesn't work!
>>
>> Aurora Log:
>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109
>> [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>
>> Thrift client:
>> ./mrfantastic_service.out
>> Connecting to aurora....
>> <nil> EOF
>>
>> Thrift client sourcec:
>> func thriftAuroraJobs() {
>> var protocolFactory thrift.TProtocolFactory
>> var transportFactory thrift.TTransportFactory
>> var transport thrift.TTransport
>> var client *api.ReadOnlySchedulerClient
>> var err error
>>
>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>> //transportFactory =
>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>> transportFactory = thrift.NewTTransportFactory()
>> fmt.Println("Connecting to aurora....")
>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>> if err != nil {
>> fmt.Println("Error opening socket:", err)
>> os.Exit(1)
>> //return err
>> }
>> if transport == nil {
>> os.Exit(1)
>> }
>> transport = transportFactory.GetTransport(transport)
>> if transport == nil {
>> os.Exit(1)
>> }
>> err = transport.Open()
>> if err != nil {
>> os.Exit(1)
>> }
>> defer transport.Close()
>> client = api.NewReadOnlySchedulerClientFactory(transport,
>> protocolFactory)
>> fmt.Println(client.GetJobs(""))
>> }
>>
>>
>> My hunch is that the thrift port isn't 8081, as the server/aurora is
>> looking for HTTP data on the socket.
>> Is there a config that needs to be set for thrift API to be initialized?
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org>
>> wrote:
>>
>>> if you just need the compiler you can install the thrift-compiler
>>> package with one of the following, otherwise you can run ./bootstrap.sh &&
>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>
>>> -Jake
>>>
>>>
>>>
>>> deb packaging (tested with ubuntu trusty)
>>>
>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>> > /etc/apt/sources.list.d/thrift.list
>>>
>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>
>>>
>>> or for centos/rhel (tested with centos 7.2)
>>>
>>> > /etc/yum.repos.d/thrift.repo
>>>
>>> [thrift]
>>> name=Apache Thrift rpm repo
>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>> enabled=1
>>> gpgcheck=0
>>>
>>>
>>>
>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Jake/Chris,
>>>> Thanks for the info.
>>>> When I try to install thrift v0.9.3 from source, I get an error as
>>>> follows while running `make check`:
>>>> ...
>>>> ...
>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>>> elapsed: 0.062 sec
>>>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>>> elapsed: 0.046 sec
>>>>
>>>> BUILD FAILED
>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>
>>>> Total time: 17 seconds
>>>> make[3]: *** [check-local] Error 1
>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>> make[2]: *** [check-am] Error 2
>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>> make[1]: *** [check-recursive] Error 1
>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>> make: *** [check-recursive] Error 1
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <c.bannister@gmail.com
>>>> > wrote:
>>>>
>>>>> I've used the latest thrift to generate go code, and then manually
>>>>> created executor config which works and is able to launch jobs.
>>>>>
>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Krish
>>>>>> We are using Thrift with go for all our api calls to Aurora, would
>>>>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>
>>>>>> happy to help answer any questions you might have
>>>>>>
>>>>>> -Jake
>>>>>>
>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks, Bill.
>>>>>>>
>>>>>>> Well I have started my foray into the the thrift API today. And I
>>>>>>> think I am stuck with some thrift configs.
>>>>>>>
>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk
>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>>>> changelog or compatibility statement on the thrift project site.
>>>>>>>
>>>>>>>
>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos
>>>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>>> krish@krish:/tmp
>>>>>>> > thrift --gen go api.thrift
>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>
>>>>>>> I can modify the golang code by hand, but I would like to play it
>>>>>>> safe and use the working compiler from the debian repos.
>>>>>>>
>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a
>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>>> similar error and gone past it?
>>>>>>> I have already done a `go get
>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Regarding documentation - Maxim is correct that there isn't much in
>>>>>>>> the way of independent/holistic docs for the thrift API. There is,
>>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>
>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>>> to that point!
>>>>>>>>
>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>> maxim@apache.org> wrote:
>>>>>>>>
>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>>>> return thrift values only in Response.result field.
>>>>>>>>>
>>>>>>>>> Correct. There is also 'details' field that may have additional
>>>>>>>>> messages (of error or informational nature)
>>>>>>>>>
>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>> understand Thrift API better?
>>>>>>>>>
>>>>>>>>> The thrift API is largely undocumented. There is an effort to
>>>>>>>>> bring up a fully supported REST API that will presumably get documented and
>>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>>
>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>
>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>> implementation to see how it happens.
>>>>>>>>>
>>>>>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>>>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>>>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>>>>>>> with the same JobKey.
>>>>>>>>>
>>>>>>>>> Ignore this object as it's an echo of the old client updater. It's
>>>>>>>>> now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>>>
>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>
>>>>>>>>> Another echo of the client updater but this time it's got a second
>>>>>>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>>>>>>> essentially a "scale-out" API that can add instances to the existing job
>>>>>>>>> without changing the underlying task assumptions.
>>>>>>>>>
>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>
>>>>>>>>> It's actually 'getPendingReason' and is currently used exclusively
>>>>>>>>> by the UI to get the reason for a task PENDING state.
>>>>>>>>>
>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>
>>>>>>>>> This is to set role-level quota. Currently only required for tasks
>>>>>>>>> with 'production=True'. Search through our docs for more details.
>>>>>>>>>
>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>> cluster.
>>>>>>>>>
>>>>>>>>> It's quite versatile and can be used to kill some or all instances
>>>>>>>>> of the job.
>>>>>>>>>
>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>
>>>>>>>>> Your observations are correct. This is the main API to change your
>>>>>>>>> service job in any way (including adding, removing or modifying instances).
>>>>>>>>>
>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>
>>>>>>>>> Scheduler will try to schedule as many instances as it can. Those
>>>>>>>>> that will not find resources will remain in PENDING state until more
>>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>>> remain PENDING.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I was going through the Aurora Thrift API to determine how to add
>>>>>>>>>> new jobs.
>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>>>>>> mesos v0.25 accordingly.
>>>>>>>>>>
>>>>>>>>>> Below is a summary of my (very limited) understanding of some
>>>>>>>>>> APIs, & would help it if someone can point out flaws in my understanding:
>>>>>>>>>>
>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>
>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>
>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>>>>> files for job configuration.
>>>>>>>>>>
>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>
>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>> By the naming convention, it seems this is used to increase
>>>>>>>>>> the number of instances of a job. It will not result in stopping of current
>>>>>>>>>> instances of the job.
>>>>>>>>>>
>>>>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>>>>
>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>> Return the reason (in string) about why the job is PENDING.
>>>>>>>>>> For example: insufficient CPU.
>>>>>>>>>>
>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>
>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>> cluster.
>>>>>>>>>>
>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>> Used for updating jobs with the new TaskConfig specified. Can
>>>>>>>>>> be used if resource requirement changes. For example: If I wanted aurora to
>>>>>>>>>> update the version of container used for a job using TaskConfig.Container
>>>>>>>>>> attribute.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> κρισhναν
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Apologies for multiple mails, the previous email was sent accidentally.
I didn't add a problem description, before hitting send.
When I query aurora for all the jobs using getJobs, I find the aurora error
as given below and the response I get using my client.
--
κρισhναν
On Fri, Mar 18, 2016 at 8:15 PM, Krish <kr...@gmail.com> wrote:
> Thanks Jake!
>
> That worked like a charm, & I was wondering why does install from source
> doesn't work!
>
> Aurora Log:
> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109
> [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>
> Thrift client:
> ./mrfantastic_service.out
> Connecting to aurora....
> <nil> EOF
>
> Thrift client sourcec:
> func thriftAuroraJobs() {
> var protocolFactory thrift.TProtocolFactory
> var transportFactory thrift.TTransportFactory
> var transport thrift.TTransport
> var client *api.ReadOnlySchedulerClient
> var err error
>
> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
> //transportFactory =
> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
> transportFactory = thrift.NewTTransportFactory()
> fmt.Println("Connecting to aurora....")
> transport, err = thrift.NewTSocket("54.210.234.190:8081")
> if err != nil {
> fmt.Println("Error opening socket:", err)
> os.Exit(1)
> //return err
> }
> if transport == nil {
> os.Exit(1)
> }
> transport = transportFactory.GetTransport(transport)
> if transport == nil {
> os.Exit(1)
> }
> err = transport.Open()
> if err != nil {
> os.Exit(1)
> }
> defer transport.Close()
> client = api.NewReadOnlySchedulerClientFactory(transport,
> protocolFactory)
> fmt.Println(client.GetJobs(""))
> }
>
>
> My hunch is that the thrift port isn't 8081, as the server/aurora is
> looking for HTTP data on the socket.
> Is there a config that needs to be set for thrift API to be initialized?
>
>
>
>
> --
> κρισhναν
>
> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org>
> wrote:
>
>> if you just need the compiler you can install the thrift-compiler package
>> with one of the following, otherwise you can run ./bootstrap.sh &&
>> ./configure && cd compiler/cpp && make to just build the compiler.
>>
>> -Jake
>>
>>
>>
>> deb packaging (tested with ubuntu trusty)
>>
>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>> > /etc/apt/sources.list.d/thrift.list
>>
>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>
>>
>> or for centos/rhel (tested with centos 7.2)
>>
>> > /etc/yum.repos.d/thrift.repo
>>
>> [thrift]
>> name=Apache Thrift rpm repo
>> baseurl=http://www.apache.org/dist/thrift/rpm/
>> enabled=1
>> gpgcheck=0
>>
>>
>>
>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com>
>> wrote:
>>
>>> Jake/Chris,
>>> Thanks for the info.
>>> When I try to install thrift v0.9.3 from source, I get an error as
>>> follows while running `make check`:
>>> ...
>>> ...
>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>> elapsed: 0.062 sec
>>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>> elapsed: 0.046 sec
>>>
>>> BUILD FAILED
>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>
>>> Total time: 17 seconds
>>> make[3]: *** [check-local] Error 1
>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>> make[2]: *** [check-am] Error 2
>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>> make[1]: *** [check-recursive] Error 1
>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>> make: *** [check-recursive] Error 1
>>>
>>>
>>>
>>>
>>> --
>>> κρισhναν
>>>
>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <c....@gmail.com>
>>> wrote:
>>>
>>>> I've used the latest thrift to generate go code, and then manually
>>>> created executor config which works and is able to launch jobs.
>>>>
>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Krish
>>>>> We are using Thrift with go for all our api calls to Aurora, would
>>>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>>>
>>>>> happy to help answer any questions you might have
>>>>>
>>>>> -Jake
>>>>>
>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks, Bill.
>>>>>>
>>>>>> Well I have started my foray into the the thrift API today. And I
>>>>>> think I am stuck with some thrift configs.
>>>>>>
>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk with
>>>>>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>>> changelog or compatibility statement on the thrift project site.
>>>>>>
>>>>>>
>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos
>>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>>> krish@krish:/tmp
>>>>>> > thrift --gen go api.thrift
>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>
>>>>>> I can modify the golang code by hand, but I would like to play it
>>>>>> safe and use the working compiler from the debian repos.
>>>>>>
>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a test
>>>>>> golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>>> similar error and gone past it?
>>>>>> I have already done a `go get
>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> κρισhναν
>>>>>>
>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Regarding documentation - Maxim is correct that there isn't much in
>>>>>>> the way of independent/holistic docs for the thrift API. There is,
>>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>
>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>> difficult API method will be defining the ExecutorConfig.data value when
>>>>>>> calling createJob. Please don't hesitate to ask for assistance if you get
>>>>>>> to that point!
>>>>>>>
>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <maxim@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>>> return thrift values only in Response.result field.
>>>>>>>>
>>>>>>>> Correct. There is also 'details' field that may have additional
>>>>>>>> messages (of error or informational nature)
>>>>>>>>
>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>> understand Thrift API better?
>>>>>>>>
>>>>>>>> The thrift API is largely undocumented. There is an effort to bring
>>>>>>>> up a fully supported REST API that will presumably get documented and
>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>
>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>
>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>> (batch) job created. The JobDescription is populated from the .aurora
>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>> implementation to see how it happens.
>>>>>>>>
>>>>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>>>>>> with the same JobKey.
>>>>>>>>
>>>>>>>> Ignore this object as it's an echo of the old client updater. It's
>>>>>>>> now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>>
>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>
>>>>>>>> Another echo of the client updater but this time it's got a second
>>>>>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>>>>>> essentially a "scale-out" API that can add instances to the existing job
>>>>>>>> without changing the underlying task assumptions.
>>>>>>>>
>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>
>>>>>>>> It's actually 'getPendingReason' and is currently used exclusively
>>>>>>>> by the UI to get the reason for a task PENDING state.
>>>>>>>>
>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>
>>>>>>>> This is to set role-level quota. Currently only required for tasks
>>>>>>>> with 'production=True'. Search through our docs for more details.
>>>>>>>>
>>>>>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>>>>>
>>>>>>>> It's quite versatile and can be used to kill some or all instances
>>>>>>>> of the job.
>>>>>>>>
>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>
>>>>>>>> Your observations are correct. This is the main API to change your
>>>>>>>> service job in any way (including adding, removing or modifying instances).
>>>>>>>>
>>>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>>>
>>>>>>>> Scheduler will try to schedule as many instances as it can. Those
>>>>>>>> that will not find resources will remain in PENDING state until more
>>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>>> remain PENDING.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I was going through the Aurora Thrift API to determine how to add
>>>>>>>>> new jobs.
>>>>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>>>>> mesos v0.25 accordingly.
>>>>>>>>>
>>>>>>>>> Below is a summary of my (very limited) understanding of some
>>>>>>>>> APIs, & would help it if someone can point out flaws in my understanding:
>>>>>>>>>
>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>
>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>> understand Thrift API better?
>>>>>>>>>
>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>>>> files for job configuration.
>>>>>>>>>
>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>
>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>> By the naming convention, it seems this is used to increase
>>>>>>>>> the number of instances of a job. It will not result in stopping of current
>>>>>>>>> instances of the job.
>>>>>>>>>
>>>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>>>
>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>> Return the reason (in string) about why the job is PENDING.
>>>>>>>>> For example: insufficient CPU.
>>>>>>>>>
>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>
>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>> cluster.
>>>>>>>>>
>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>> Used for updating jobs with the new TaskConfig specified. Can
>>>>>>>>> be used if resource requirement changes. For example: If I wanted aurora to
>>>>>>>>> update the version of container used for a job using TaskConfig.Container
>>>>>>>>> attribute.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>> instances, and there are resources available to run only 4 of them, does
>>>>>>>>> the entire job block, or only the 5th instance of the job blocks?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> κρισhναν
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Thanks Jake!
That worked like a charm, & I was wondering why does install from source
doesn't work!
Aurora Log:
Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109
[qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
Thrift client:
./mrfantastic_service.out
Connecting to aurora....
<nil> EOF
Thrift client sourcec:
func thriftAuroraJobs() {
var protocolFactory thrift.TProtocolFactory
var transportFactory thrift.TTransportFactory
var transport thrift.TTransport
var client *api.ReadOnlySchedulerClient
var err error
protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
//transportFactory =
thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
transportFactory = thrift.NewTTransportFactory()
fmt.Println("Connecting to aurora....")
transport, err = thrift.NewTSocket("54.210.234.190:8081")
if err != nil {
fmt.Println("Error opening socket:", err)
os.Exit(1)
//return err
}
if transport == nil {
os.Exit(1)
}
transport = transportFactory.GetTransport(transport)
if transport == nil {
os.Exit(1)
}
err = transport.Open()
if err != nil {
os.Exit(1)
}
defer transport.Close()
client = api.NewReadOnlySchedulerClientFactory(transport,
protocolFactory)
fmt.Println(client.GetJobs(""))
}
My hunch is that the thrift port isn't 8081, as the server/aurora is
looking for HTTP data on the socket.
Is there a config that needs to be set for thrift API to be initialized?
--
κρισhναν
On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <jf...@apache.org> wrote:
> if you just need the compiler you can install the thrift-compiler package
> with one of the following, otherwise you can run ./bootstrap.sh &&
> ./configure && cd compiler/cpp && make to just build the compiler.
>
> -Jake
>
>
>
> deb packaging (tested with ubuntu trusty)
>
> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
> > gpg --export --armor 66B778F9 | sudo apt-key add -
> > /etc/apt/sources.list.d/thrift.list
>
> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>
>
> or for centos/rhel (tested with centos 7.2)
>
> > /etc/yum.repos.d/thrift.repo
>
> [thrift]
> name=Apache Thrift rpm repo
> baseurl=http://www.apache.org/dist/thrift/rpm/
> enabled=1
> gpgcheck=0
>
>
>
> On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com> wrote:
>
>> Jake/Chris,
>> Thanks for the info.
>> When I try to install thrift v0.9.3 from source, I get an error as
>> follows while running `make check`:
>> ...
>> ...
>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>> elapsed: 0.062 sec
>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>> elapsed: 0.046 sec
>>
>> BUILD FAILED
>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>
>> Total time: 17 seconds
>> make[3]: *** [check-local] Error 1
>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>> make[2]: *** [check-am] Error 2
>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>> make[1]: *** [check-recursive] Error 1
>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>> make: *** [check-recursive] Error 1
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <c....@gmail.com>
>> wrote:
>>
>>> I've used the latest thrift to generate go code, and then manually
>>> created executor config which works and is able to launch jobs.
>>>
>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org>
>>> wrote:
>>>
>>>> Hi Krish
>>>> We are using Thrift with go for all our api calls to Aurora, would
>>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>>
>>>> happy to help answer any questions you might have
>>>>
>>>> -Jake
>>>>
>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks, Bill.
>>>>>
>>>>> Well I have started my foray into the the thrift API today. And I
>>>>> think I am stuck with some thrift configs.
>>>>>
>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk with
>>>>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>> changelog or compatibility statement on the thrift project site.
>>>>>
>>>>>
>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos
>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>> krish@krish:/tmp
>>>>> > thrift --gen go api.thrift
>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>
>>>>> I can modify the golang code by hand, but I would like to play it safe
>>>>> and use the working compiler from the debian repos.
>>>>>
>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a test
>>>>> golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>> similar error and gone past it?
>>>>> I have already done a `go get
>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Regarding documentation - Maxim is correct that there isn't much in
>>>>>> the way of independent/holistic docs for the thrift API. There is,
>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>
>>>>>> If you are looking to use the thrift API directly, the most difficult
>>>>>> API method will be defining the ExecutorConfig.data value when calling
>>>>>> createJob. Please don't hesitate to ask for assistance if you get to that
>>>>>> point!
>>>>>>
>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>> return thrift values only in Response.result field.
>>>>>>>
>>>>>>> Correct. There is also 'details' field that may have additional
>>>>>>> messages (of error or informational nature)
>>>>>>>
>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>> understand Thrift API better?
>>>>>>>
>>>>>>> The thrift API is largely undocumented. There is an effort to bring
>>>>>>> up a fully supported REST API that will presumably get documented and
>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>
>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>
>>>>>>> This is the API to use when you a brand new service or adhoc (batch)
>>>>>>> job created. The JobDescription is populated from the .aurora config. You
>>>>>>> may want to trace "aurora job create" client command implementation to see
>>>>>>> how it happens.
>>>>>>>
>>>>>>> 4. What is the Lock object? I see that some APIs require locking and
>>>>>>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>>>>>>> assuming that it is required so that one does not create multiple jobs with
>>>>>>>> the same JobKey.
>>>>>>>
>>>>>>> Ignore this object as it's an echo of the old client updater. It's
>>>>>>> now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>
>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>
>>>>>>> Another echo of the client updater but this time it's got a second
>>>>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>>>>> essentially a "scale-out" API that can add instances to the existing job
>>>>>>> without changing the underlying task assumptions.
>>>>>>>
>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>
>>>>>>> It's actually 'getPendingReason' and is currently used exclusively
>>>>>>> by the UI to get the reason for a task PENDING state.
>>>>>>>
>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>
>>>>>>> This is to set role-level quota. Currently only required for tasks
>>>>>>> with 'production=True'. Search through our docs for more details.
>>>>>>>
>>>>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>>>>
>>>>>>> It's quite versatile and can be used to kill some or all instances
>>>>>>> of the job.
>>>>>>>
>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>
>>>>>>> Your observations are correct. This is the main API to change your
>>>>>>> service job in any way (including adding, removing or modifying instances).
>>>>>>>
>>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>>
>>>>>>> Scheduler will try to schedule as many instances as it can. Those
>>>>>>> that will not find resources will remain in PENDING state until more
>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>> remain PENDING.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I was going through the Aurora Thrift API to determine how to add
>>>>>>>> new jobs.
>>>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>>>> mesos v0.25 accordingly.
>>>>>>>>
>>>>>>>> Below is a summary of my (very limited) understanding of some APIs,
>>>>>>>> & would help it if someone can point out flaws in my understanding:
>>>>>>>>
>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>> return thrift values only in Response.result field.
>>>>>>>>
>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>> understand Thrift API better?
>>>>>>>>
>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>>> files for job configuration.
>>>>>>>>
>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>> locking and some don't. For example, createJob needs a Lock object as
>>>>>>>> parameter, & I am assuming that it is required so that one does not create
>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>
>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>> By the naming convention, it seems this is used to increase the
>>>>>>>> number of instances of a job. It will not result in stopping of current
>>>>>>>> instances of the job.
>>>>>>>>
>>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>>
>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>> Return the reason (in string) about why the job is PENDING. For
>>>>>>>> example: insufficient CPU.
>>>>>>>>
>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>
>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>> cluster.
>>>>>>>>
>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>> Used for updating jobs with the new TaskConfig specified. Can
>>>>>>>> be used if resource requirement changes. For example: If I wanted aurora to
>>>>>>>> update the version of container used for a job using TaskConfig.Container
>>>>>>>> attribute.
>>>>>>>>
>>>>>>>>
>>>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> --
>>>>>>>> κρισhναν
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
Re: Aurora Thrift API Info
Posted by Jake Farrell <jf...@apache.org>.
if you just need the compiler you can install the thrift-compiler package
with one of the following, otherwise you can run ./bootstrap.sh &&
./configure && cd compiler/cpp && make to just build the compiler.
-Jake
deb packaging (tested with ubuntu trusty)
> curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
> gpg --export --armor 66B778F9 | sudo apt-key add -
> /etc/apt/sources.list.d/thrift.list
deb http://www.apache.org/dist/thrift/debian 0.9.3 main
or for centos/rhel (tested with centos 7.2)
> /etc/yum.repos.d/thrift.repo
[thrift]
name=Apache Thrift rpm repo
baseurl=http://www.apache.org/dist/thrift/rpm/
enabled=1
gpgcheck=0
On Thu, Mar 17, 2016 at 12:32 PM, Krish <kr...@gmail.com> wrote:
> Jake/Chris,
> Thanks for the info.
> When I try to install thrift v0.9.3 from source, I get an error as follows
> while running `make check`:
> ...
> ...
> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 0.062 sec
> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 0.046 sec
>
> BUILD FAILED
> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>
> Total time: 17 seconds
> make[3]: *** [check-local] Error 1
> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
> make[1]: *** [check-recursive] Error 1
> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
> make: *** [check-recursive] Error 1
>
>
>
>
> --
> κρισhναν
>
> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <c....@gmail.com>
> wrote:
>
>> I've used the latest thrift to generate go code, and then manually
>> created executor config which works and is able to launch jobs.
>>
>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org> wrote:
>>
>>> Hi Krish
>>> We are using Thrift with go for all our api calls to Aurora, would
>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>
>>> happy to help answer any questions you might have
>>>
>>> -Jake
>>>
>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Thanks, Bill.
>>>>
>>>> Well I have started my foray into the the thrift API today. And I think
>>>> I am stuck with some thrift configs.
>>>>
>>>> Does it matter if I use thrift v0.9.0 on the client side to talk with
>>>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>> changelog or compatibility statement on the thrift project site.
>>>>
>>>>
>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos have
>>>> 0.9.0, I had to compile the thrift compiler v0.9.1 from source. However,
>>>> when I try to generate golang code, I think I hit a compiler bug:
>>>> krish@krish:/tmp
>>>> > thrift --gen go api.thrift
>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>
>>>> I can modify the golang code by hand, but I would like to play it safe
>>>> and use the working compiler from the debian repos.
>>>>
>>>> Also, when I use thrift v0.9.0, and try to integrate code into a test
>>>> golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>> similar error and gone past it?
>>>> I have already done a `go get
>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>>> wrote:
>>>>
>>>>> Regarding documentation - Maxim is correct that there isn't much in
>>>>> the way of independent/holistic docs for the thrift API. There is,
>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>
>>>>> If you are looking to use the thrift API directly, the most difficult
>>>>> API method will be defining the ExecutorConfig.data value when calling
>>>>> createJob. Please don't hesitate to ask for assistance if you get to that
>>>>> point!
>>>>>
>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>> return thrift values only in Response.result field.
>>>>>>
>>>>>> Correct. There is also 'details' field that may have additional
>>>>>> messages (of error or informational nature)
>>>>>>
>>>>>> 2. Is there a set of examples in the documentation to help understand
>>>>>>> Thrift API better?
>>>>>>
>>>>>> The thrift API is largely undocumented. There is an effort to bring
>>>>>> up a fully supported REST API that will presumably get documented and
>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>
>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>
>>>>>> This is the API to use when you a brand new service or adhoc (batch)
>>>>>> job created. The JobDescription is populated from the .aurora config. You
>>>>>> may want to trace "aurora job create" client command implementation to see
>>>>>> how it happens.
>>>>>>
>>>>>> 4. What is the Lock object? I see that some APIs require locking and
>>>>>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>>>>>> assuming that it is required so that one does not create multiple jobs with
>>>>>>> the same JobKey.
>>>>>>
>>>>>> Ignore this object as it's an echo of the old client updater. It's
>>>>>> now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>
>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>
>>>>>> Another echo of the client updater but this time it's got a second
>>>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>>>> essentially a "scale-out" API that can add instances to the existing job
>>>>>> without changing the underlying task assumptions.
>>>>>>
>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>
>>>>>> It's actually 'getPendingReason' and is currently used exclusively by
>>>>>> the UI to get the reason for a task PENDING state.
>>>>>>
>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>
>>>>>> This is to set role-level quota. Currently only required for tasks
>>>>>> with 'production=True'. Search through our docs for more details.
>>>>>>
>>>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>>>
>>>>>> It's quite versatile and can be used to kill some or all instances of
>>>>>> the job.
>>>>>>
>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>
>>>>>> Your observations are correct. This is the main API to change your
>>>>>> service job in any way (including adding, removing or modifying instances).
>>>>>>
>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>
>>>>>> Scheduler will try to schedule as many instances as it can. Those
>>>>>> that will not find resources will remain in PENDING state until more
>>>>>> resources are available. In your particular example only the 5th will
>>>>>> remain PENDING.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I was going through the Aurora Thrift API to determine how to add
>>>>>>> new jobs.
>>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>>> mesos v0.25 accordingly.
>>>>>>>
>>>>>>> Below is a summary of my (very limited) understanding of some APIs,
>>>>>>> & would help it if someone can point out flaws in my understanding:
>>>>>>>
>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>> return thrift values only in Response.result field.
>>>>>>>
>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>> understand Thrift API better?
>>>>>>>
>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>> files for job configuration.
>>>>>>>
>>>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>>>> with the same JobKey.
>>>>>>>
>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>> By the naming convention, it seems this is used to increase the
>>>>>>> number of instances of a job. It will not result in stopping of current
>>>>>>> instances of the job.
>>>>>>>
>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>
>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>> Return the reason (in string) about why the job is PENDING. For
>>>>>>> example: insufficient CPU.
>>>>>>>
>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>
>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>> cluster.
>>>>>>>
>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>> Used for updating jobs with the new TaskConfig specified. Can be
>>>>>>> used if resource requirement changes. For example: If I wanted aurora to
>>>>>>> update the version of container used for a job using TaskConfig.Container
>>>>>>> attribute.
>>>>>>>
>>>>>>>
>>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Jake/Chris,
Thanks for the info.
When I try to install thrift v0.9.3 from source, I get an error as follows
while running `make check`:
...
...
[junit] Running org.apache.thrift.protocol.TestTProtocolUtil
[junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.062 sec
[junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
[junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.046 sec
BUILD FAILED
/tmp/thrift-0.9.3/lib/java/build.xml:202: Test
org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
Total time: 17 seconds
make[3]: *** [check-local] Error 1
make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
make: *** [check-recursive] Error 1
--
κρισhναν
On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <c....@gmail.com>
wrote:
> I've used the latest thrift to generate go code, and then manually created
> executor config which works and is able to launch jobs.
>
> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org> wrote:
>
>> Hi Krish
>> We are using Thrift with go for all our api calls to Aurora, would
>> recommend you use Thrift 0.9.3 to interact with the api.
>>
>> happy to help answer any questions you might have
>>
>> -Jake
>>
>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com> wrote:
>>
>>> Thanks, Bill.
>>>
>>> Well I have started my foray into the the thrift API today. And I think
>>> I am stuck with some thrift configs.
>>>
>>> Does it matter if I use thrift v0.9.0 on the client side to talk with
>>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>> changelog or compatibility statement on the thrift project site.
>>>
>>>
>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos have
>>> 0.9.0, I had to compile the thrift compiler v0.9.1 from source. However,
>>> when I try to generate golang code, I think I hit a compiler bug:
>>> krish@krish:/tmp
>>> > thrift --gen go api.thrift
>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>
>>> I can modify the golang code by hand, but I would like to play it safe
>>> and use the working compiler from the debian repos.
>>>
>>> Also, when I use thrift v0.9.0, and try to integrate code into a test
>>> golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>> similar error and gone past it?
>>> I have already done a `go get
>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>
>>>
>>>
>>>
>>> --
>>> κρισhναν
>>>
>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org>
>>> wrote:
>>>
>>>> Regarding documentation - Maxim is correct that there isn't much in the
>>>> way of independent/holistic docs for the thrift API. There is, however,
>>>> scant javadoc-style documentation within the IDL spec itself:
>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>
>>>> If you are looking to use the thrift API directly, the most difficult
>>>> API method will be defining the ExecutorConfig.data value when calling
>>>> createJob. Please don't hesitate to ask for assistance if you get to that
>>>> point!
>>>>
>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>>>> wrote:
>>>>
>>>>> 1. All APIs require thrift inputs of the structs specified, and return
>>>>>> thrift values only in Response.result field.
>>>>>
>>>>> Correct. There is also 'details' field that may have additional
>>>>> messages (of error or informational nature)
>>>>>
>>>>> 2. Is there a set of examples in the documentation to help understand
>>>>>> Thrift API better?
>>>>>
>>>>> The thrift API is largely undocumented. There is an effort to bring up
>>>>> a fully supported REST API that will presumably get documented and become
>>>>> much easier to use. It's mostly in flux now.
>>>>>
>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>
>>>>> This is the API to use when you a brand new service or adhoc (batch)
>>>>> job created. The JobDescription is populated from the .aurora config. You
>>>>> may want to trace "aurora job create" client command implementation to see
>>>>> how it happens.
>>>>>
>>>>> 4. What is the Lock object? I see that some APIs require locking and
>>>>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>>>>> assuming that it is required so that one does not create multiple jobs with
>>>>>> the same JobKey.
>>>>>
>>>>> Ignore this object as it's an echo of the old client updater. It's now
>>>>> deprecated and will be removed soon. You can pass NULL for now.
>>>>>
>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>
>>>>> Another echo of the client updater but this time it's got a second
>>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>>> essentially a "scale-out" API that can add instances to the existing job
>>>>> without changing the underlying task assumptions.
>>>>>
>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>
>>>>> It's actually 'getPendingReason' and is currently used exclusively by
>>>>> the UI to get the reason for a task PENDING state.
>>>>>
>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>
>>>>> This is to set role-level quota. Currently only required for tasks
>>>>> with 'production=True'. Search through our docs for more details.
>>>>>
>>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>>
>>>>> It's quite versatile and can be used to kill some or all instances of
>>>>> the job.
>>>>>
>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>
>>>>> Your observations are correct. This is the main API to change your
>>>>> service job in any way (including adding, removing or modifying instances).
>>>>>
>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>
>>>>> Scheduler will try to schedule as many instances as it can. Those that
>>>>> will not find resources will remain in PENDING state until more resources
>>>>> are available. In your particular example only the 5th will remain PENDING.
>>>>>
>>>>>
>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I was going through the Aurora Thrift API to determine how to add new
>>>>>> jobs.
>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>> mesos v0.25 accordingly.
>>>>>>
>>>>>> Below is a summary of my (very limited) understanding of some APIs, &
>>>>>> would help it if someone can point out flaws in my understanding:
>>>>>>
>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>> return thrift values only in Response.result field.
>>>>>>
>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>> understand Thrift API better?
>>>>>>
>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>> This is basically the API to replace the Aurora DSL/.aurora files
>>>>>> for job configuration.
>>>>>>
>>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>>> with the same JobKey.
>>>>>>
>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>> By the naming convention, it seems this is used to increase the
>>>>>> number of instances of a job. It will not result in stopping of current
>>>>>> instances of the job.
>>>>>>
>>>>>> My second explanation for this API: Since it uses a set of
>>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>>> internal data structures of Aurora to track the job.
>>>>>>
>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>> Return the reason (in string) about why the job is PENDING. For
>>>>>> example: insufficient CPU.
>>>>>>
>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>
>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>> cluster.
>>>>>>
>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>> Used for updating jobs with the new TaskConfig specified. Can be
>>>>>> used if resource requirement changes. For example: If I wanted aurora to
>>>>>> update the version of container used for a job using TaskConfig.Container
>>>>>> attribute.
>>>>>>
>>>>>>
>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>> and there are resources available to run only 4 of them, does the entire
>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> --
>>>>>> κρισhναν
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Aurora Thrift API Info
Posted by Chris Bannister <c....@gmail.com>.
I've used the latest thrift to generate go code, and then manually created
executor config which works and is able to launch jobs.
On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <jf...@apache.org> wrote:
> Hi Krish
> We are using Thrift with go for all our api calls to Aurora, would
> recommend you use Thrift 0.9.3 to interact with the api.
>
> happy to help answer any questions you might have
>
> -Jake
>
> On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com> wrote:
>
>> Thanks, Bill.
>>
>> Well I have started my foray into the the thrift API today. And I think I
>> am stuck with some thrift configs.
>>
>> Does it matter if I use thrift v0.9.0 on the client side to talk with
>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>> changelog or compatibility statement on the thrift project site.
>>
>>
>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos have
>> 0.9.0, I had to compile the thrift compiler v0.9.1 from source. However,
>> when I try to generate golang code, I think I hit a compiler bug:
>> krish@krish:/tmp
>> > thrift --gen go api.thrift
>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>
>> I can modify the golang code by hand, but I would like to play it safe
>> and use the working compiler from the debian repos.
>>
>> Also, when I use thrift v0.9.0, and try to integrate code into a test
>> golang app, it fails to find "thriftlib/api" package. Anyone faced a
>> similar error and gone past it?
>> I have already done a `go get
>> git.apache.org/thrift.git/lib/go/thrift/...`
>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org> wrote:
>>
>>> Regarding documentation - Maxim is correct that there isn't much in the
>>> way of independent/holistic docs for the thrift API. There is, however,
>>> scant javadoc-style documentation within the IDL spec itself:
>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>
>>> If you are looking to use the thrift API directly, the most difficult
>>> API method will be defining the ExecutorConfig.data value when calling
>>> createJob. Please don't hesitate to ask for assistance if you get to that
>>> point!
>>>
>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>>> wrote:
>>>
>>>> 1. All APIs require thrift inputs of the structs specified, and return
>>>>> thrift values only in Response.result field.
>>>>
>>>> Correct. There is also 'details' field that may have additional
>>>> messages (of error or informational nature)
>>>>
>>>> 2. Is there a set of examples in the documentation to help understand
>>>>> Thrift API better?
>>>>
>>>> The thrift API is largely undocumented. There is an effort to bring up
>>>> a fully supported REST API that will presumably get documented and become
>>>> much easier to use. It's mostly in flux now.
>>>>
>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>
>>>> This is the API to use when you a brand new service or adhoc (batch)
>>>> job created. The JobDescription is populated from the .aurora config. You
>>>> may want to trace "aurora job create" client command implementation to see
>>>> how it happens.
>>>>
>>>> 4. What is the Lock object? I see that some APIs require locking and
>>>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>>>> assuming that it is required so that one does not create multiple jobs with
>>>>> the same JobKey.
>>>>
>>>> Ignore this object as it's an echo of the old client updater. It's now
>>>> deprecated and will be removed soon. You can pass NULL for now.
>>>>
>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>
>>>> Another echo of the client updater but this time it's got a second
>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>> essentially a "scale-out" API that can add instances to the existing job
>>>> without changing the underlying task assumptions.
>>>>
>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>
>>>> It's actually 'getPendingReason' and is currently used exclusively by
>>>> the UI to get the reason for a task PENDING state.
>>>>
>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>
>>>> This is to set role-level quota. Currently only required for tasks with
>>>> 'production=True'. Search through our docs for more details.
>>>>
>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>
>>>> It's quite versatile and can be used to kill some or all instances of
>>>> the job.
>>>>
>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>
>>>> Your observations are correct. This is the main API to change your
>>>> service job in any way (including adding, removing or modifying instances).
>>>>
>>>> An aurora scheduling question is if I start a job with 5 instances, and
>>>>> there are resources available to run only 4 of them, does the entire job
>>>>> block, or only the 5th instance of the job blocks?
>>>>
>>>> Scheduler will try to schedule as many instances as it can. Those that
>>>> will not find resources will remain in PENDING state until more resources
>>>> are available. In your particular example only the 5th will remain PENDING.
>>>>
>>>>
>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> I was going through the Aurora Thrift API to determine how to add new
>>>>> jobs.
>>>>> I am using aurora v0.12 released last month and have upgraded to mesos
>>>>> v0.25 accordingly.
>>>>>
>>>>> Below is a summary of my (very limited) understanding of some APIs, &
>>>>> would help it if someone can point out flaws in my understanding:
>>>>>
>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>> return thrift values only in Response.result field.
>>>>>
>>>>> 2. Is there a set of examples in the documentation to help
>>>>> understand Thrift API better?
>>>>>
>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>> This is basically the API to replace the Aurora DSL/.aurora files
>>>>> for job configuration.
>>>>>
>>>>> 4. What is the Lock object? I see that some APIs require locking
>>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>>> I am assuming that it is required so that one does not create multiple jobs
>>>>> with the same JobKey.
>>>>>
>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>> By the naming convention, it seems this is used to increase the
>>>>> number of instances of a job. It will not result in stopping of current
>>>>> instances of the job.
>>>>>
>>>>> My second explanation for this API: Since it uses a set of
>>>>> instanceIds, this is used for adding already running job in slaves to the
>>>>> internal data structures of Aurora to track the job.
>>>>>
>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>> Return the reason (in string) about why the job is PENDING. For
>>>>> example: insufficient CPU.
>>>>>
>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>
>>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>>
>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>> Used for updating jobs with the new TaskConfig specified. Can be
>>>>> used if resource requirement changes. For example: If I wanted aurora to
>>>>> update the version of container used for a job using TaskConfig.Container
>>>>> attribute.
>>>>>
>>>>>
>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>> and there are resources available to run only 4 of them, does the entire
>>>>> job block, or only the 5th instance of the job blocks?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Jake Farrell <jf...@apache.org>.
Hi Krish
We are using Thrift with go for all our api calls to Aurora, would
recommend you use Thrift 0.9.3 to interact with the api.
happy to help answer any questions you might have
-Jake
On Thu, Mar 17, 2016 at 9:43 AM, Krish <kr...@gmail.com> wrote:
> Thanks, Bill.
>
> Well I have started my foray into the the thrift API today. And I think I
> am stuck with some thrift configs.
>
> Does it matter if I use thrift v0.9.0 on the client side to talk with
> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
> changelog or compatibility statement on the thrift project site.
>
>
> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos have
> 0.9.0, I had to compile the thrift compiler v0.9.1 from source. However,
> when I try to generate golang code, I think I hit a compiler bug:
> krish@krish:/tmp
> > thrift --gen go api.thrift
> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>
> I can modify the golang code by hand, but I would like to play it safe and
> use the working compiler from the debian repos.
>
> Also, when I use thrift v0.9.0, and try to integrate code into a test
> golang app, it fails to find "thriftlib/api" package. Anyone faced a
> similar error and gone past it?
> I have already done a `go get git.apache.org/thrift.git/lib/go/thrift/...`
> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>
>
>
>
> --
> κρισhναν
>
> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org> wrote:
>
>> Regarding documentation - Maxim is correct that there isn't much in the
>> way of independent/holistic docs for the thrift API. There is, however,
>> scant javadoc-style documentation within the IDL spec itself:
>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>
>> If you are looking to use the thrift API directly, the most difficult API
>> method will be defining the ExecutorConfig.data value when calling
>> createJob. Please don't hesitate to ask for assistance if you get to that
>> point!
>>
>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>> wrote:
>>
>>> 1. All APIs require thrift inputs of the structs specified, and return
>>>> thrift values only in Response.result field.
>>>
>>> Correct. There is also 'details' field that may have additional messages
>>> (of error or informational nature)
>>>
>>> 2. Is there a set of examples in the documentation to help understand
>>>> Thrift API better?
>>>
>>> The thrift API is largely undocumented. There is an effort to bring up a
>>> fully supported REST API that will presumably get documented and become
>>> much easier to use. It's mostly in flux now.
>>>
>>> 3. createJob(JobDescription desc, Lock lock):
>>>
>>> This is the API to use when you a brand new service or adhoc (batch) job
>>> created. The JobDescription is populated from the .aurora config. You may
>>> want to trace "aurora job create" client command implementation to see how
>>> it happens.
>>>
>>> 4. What is the Lock object? I see that some APIs require locking and
>>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>>> assuming that it is required so that one does not create multiple jobs with
>>>> the same JobKey.
>>>
>>> Ignore this object as it's an echo of the old client updater. It's now
>>> deprecated and will be removed soon. You can pass NULL for now.
>>>
>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>
>>> Another echo of the client updater but this time it's got a second life.
>>> Check out its new signature and comments in the api.thrift. It's
>>> essentially a "scale-out" API that can add instances to the existing job
>>> without changing the underlying task assumptions.
>>>
>>> 6. getPendingResult(TaskQuery taskquery):
>>>
>>> It's actually 'getPendingReason' and is currently used exclusively by
>>> the UI to get the reason for a task PENDING state.
>>>
>>> 7. setQuota & getQuota for setting user level quotas.
>>>
>>> This is to set role-level quota. Currently only required for tasks with
>>> 'production=True'. Search through our docs for more details.
>>>
>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>
>>> It's quite versatile and can be used to kill some or all instances of
>>> the job.
>>>
>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>
>>> Your observations are correct. This is the main API to change your
>>> service job in any way (including adding, removing or modifying instances).
>>>
>>> An aurora scheduling question is if I start a job with 5 instances, and
>>>> there are resources available to run only 4 of them, does the entire job
>>>> block, or only the 5th instance of the job blocks?
>>>
>>> Scheduler will try to schedule as many instances as it can. Those that
>>> will not find resources will remain in PENDING state until more resources
>>> are available. In your particular example only the 5th will remain PENDING.
>>>
>>>
>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I was going through the Aurora Thrift API to determine how to add new
>>>> jobs.
>>>> I am using aurora v0.12 released last month and have upgraded to mesos
>>>> v0.25 accordingly.
>>>>
>>>> Below is a summary of my (very limited) understanding of some APIs, &
>>>> would help it if someone can point out flaws in my understanding:
>>>>
>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>> return thrift values only in Response.result field.
>>>>
>>>> 2. Is there a set of examples in the documentation to help
>>>> understand Thrift API better?
>>>>
>>>> 3. createJob(JobDescription desc, Lock lock):
>>>> This is basically the API to replace the Aurora DSL/.aurora files
>>>> for job configuration.
>>>>
>>>> 4. What is the Lock object? I see that some APIs require locking
>>>> and some don't. For example, createJob needs a Lock object as parameter, &
>>>> I am assuming that it is required so that one does not create multiple jobs
>>>> with the same JobKey.
>>>>
>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>> By the naming convention, it seems this is used to increase the
>>>> number of instances of a job. It will not result in stopping of current
>>>> instances of the job.
>>>>
>>>> My second explanation for this API: Since it uses a set of
>>>> instanceIds, this is used for adding already running job in slaves to the
>>>> internal data structures of Aurora to track the job.
>>>>
>>>> 6. getPendingResult(TaskQuery taskquery):
>>>> Return the reason (in string) about why the job is PENDING. For
>>>> example: insufficient CPU.
>>>>
>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>
>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>
>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>> Used for updating jobs with the new TaskConfig specified. Can be
>>>> used if resource requirement changes. For example: If I wanted aurora to
>>>> update the version of container used for a job using TaskConfig.Container
>>>> attribute.
>>>>
>>>>
>>>> An aurora scheduling question is if I start a job with 5 instances, and
>>>> there are resources available to run only 4 of them, does the entire job
>>>> block, or only the 5th instance of the job blocks?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>
>>>
>>
>
Re: Aurora Thrift API Info
Posted by Krish <kr...@gmail.com>.
Thanks, Bill.
Well I have started my foray into the the thrift API today. And I think I
am stuck with some thrift configs.
Does it matter if I use thrift v0.9.0 on the client side to talk with
aurora using thrift 0.9.1? Are they compatible? I couldn't find any
changelog or compatibility statement on the thrift project site.
Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos have
0.9.0, I had to compile the thrift compiler v0.9.1 from source. However,
when I try to generate golang code, I think I hit a compiler bug:
krish@krish:/tmp
> thrift --gen go api.thrift
./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
./gen-go//api/ttypes.go:2624:19: expected '==', found '='
WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
I can modify the golang code by hand, but I would like to play it safe and
use the working compiler from the debian repos.
Also, when I use thrift v0.9.0, and try to integrate code into a test
golang app, it fails to find "thriftlib/api" package. Anyone faced a
similar error and gone past it?
I have already done a `go get git.apache.org/thrift.git/lib/go/thrift/...`
--
κρισhναν
On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <wf...@apache.org> wrote:
> Regarding documentation - Maxim is correct that there isn't much in the
> way of independent/holistic docs for the thrift API. There is, however,
> scant javadoc-style documentation within the IDL spec itself:
> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>
> If you are looking to use the thrift API directly, the most difficult API
> method will be defining the ExecutorConfig.data value when calling
> createJob. Please don't hesitate to ask for assistance if you get to that
> point!
>
> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
>
>> 1. All APIs require thrift inputs of the structs specified, and return
>>> thrift values only in Response.result field.
>>
>> Correct. There is also 'details' field that may have additional messages
>> (of error or informational nature)
>>
>> 2. Is there a set of examples in the documentation to help understand
>>> Thrift API better?
>>
>> The thrift API is largely undocumented. There is an effort to bring up a
>> fully supported REST API that will presumably get documented and become
>> much easier to use. It's mostly in flux now.
>>
>> 3. createJob(JobDescription desc, Lock lock):
>>
>> This is the API to use when you a brand new service or adhoc (batch) job
>> created. The JobDescription is populated from the .aurora config. You may
>> want to trace "aurora job create" client command implementation to see how
>> it happens.
>>
>> 4. What is the Lock object? I see that some APIs require locking and some
>>> don't. For example, createJob needs a Lock object as parameter, & I am
>>> assuming that it is required so that one does not create multiple jobs with
>>> the same JobKey.
>>
>> Ignore this object as it's an echo of the old client updater. It's now
>> deprecated and will be removed soon. You can pass NULL for now.
>>
>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>
>> Another echo of the client updater but this time it's got a second life.
>> Check out its new signature and comments in the api.thrift. It's
>> essentially a "scale-out" API that can add instances to the existing job
>> without changing the underlying task assumptions.
>>
>> 6. getPendingResult(TaskQuery taskquery):
>>
>> It's actually 'getPendingReason' and is currently used exclusively by the
>> UI to get the reason for a task PENDING state.
>>
>> 7. setQuota & getQuota for setting user level quotas.
>>
>> This is to set role-level quota. Currently only required for tasks with
>> 'production=True'. Search through our docs for more details.
>>
>> 8. killTasks to kill all running instances of a job in the cluster.
>>
>> It's quite versatile and can be used to kill some or all instances of the
>> job.
>>
>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>
>> Your observations are correct. This is the main API to change your
>> service job in any way (including adding, removing or modifying instances).
>>
>> An aurora scheduling question is if I start a job with 5 instances, and
>>> there are resources available to run only 4 of them, does the entire job
>>> block, or only the 5th instance of the job blocks?
>>
>> Scheduler will try to schedule as many instances as it can. Those that
>> will not find resources will remain in PENDING state until more resources
>> are available. In your particular example only the 5th will remain PENDING.
>>
>>
>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com> wrote:
>>
>>> Hi,
>>> I was going through the Aurora Thrift API to determine how to add new
>>> jobs.
>>> I am using aurora v0.12 released last month and have upgraded to mesos
>>> v0.25 accordingly.
>>>
>>> Below is a summary of my (very limited) understanding of some APIs, &
>>> would help it if someone can point out flaws in my understanding:
>>>
>>> 1. All APIs require thrift inputs of the structs specified, and
>>> return thrift values only in Response.result field.
>>>
>>> 2. Is there a set of examples in the documentation to help
>>> understand Thrift API better?
>>>
>>> 3. createJob(JobDescription desc, Lock lock):
>>> This is basically the API to replace the Aurora DSL/.aurora files
>>> for job configuration.
>>>
>>> 4. What is the Lock object? I see that some APIs require locking and
>>> some don't. For example, createJob needs a Lock object as parameter, & I am
>>> assuming that it is required so that one does not create multiple jobs with
>>> the same JobKey.
>>>
>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>> By the naming convention, it seems this is used to increase the
>>> number of instances of a job. It will not result in stopping of current
>>> instances of the job.
>>>
>>> My second explanation for this API: Since it uses a set of
>>> instanceIds, this is used for adding already running job in slaves to the
>>> internal data structures of Aurora to track the job.
>>>
>>> 6. getPendingResult(TaskQuery taskquery):
>>> Return the reason (in string) about why the job is PENDING. For
>>> example: insufficient CPU.
>>>
>>> 7. setQuota & getQuota for setting user level quotas.
>>>
>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>
>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>> Used for updating jobs with the new TaskConfig specified. Can be
>>> used if resource requirement changes. For example: If I wanted aurora to
>>> update the version of container used for a job using TaskConfig.Container
>>> attribute.
>>>
>>>
>>> An aurora scheduling question is if I start a job with 5 instances, and
>>> there are resources available to run only 4 of them, does the entire job
>>> block, or only the 5th instance of the job blocks?
>>>
>>> Thanks!
>>>
>>> --
>>> κρισhναν
>>>
>>
>>
>
Re: Aurora Thrift API Info
Posted by Bill Farner <wf...@apache.org>.
Regarding documentation - Maxim is correct that there isn't much in the way
of independent/holistic docs for the thrift API. There is, however, scant
javadoc-style documentation within the IDL spec itself:
https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
If you are looking to use the thrift API directly, the most difficult API
method will be defining the ExecutorConfig.data value when calling
createJob. Please don't hesitate to ask for assistance if you get to that
point!
On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <ma...@apache.org> wrote:
> 1. All APIs require thrift inputs of the structs specified, and return
>> thrift values only in Response.result field.
>
> Correct. There is also 'details' field that may have additional messages
> (of error or informational nature)
>
> 2. Is there a set of examples in the documentation to help understand
>> Thrift API better?
>
> The thrift API is largely undocumented. There is an effort to bring up a
> fully supported REST API that will presumably get documented and become
> much easier to use. It's mostly in flux now.
>
> 3. createJob(JobDescription desc, Lock lock):
>
> This is the API to use when you a brand new service or adhoc (batch) job
> created. The JobDescription is populated from the .aurora config. You may
> want to trace "aurora job create" client command implementation to see how
> it happens.
>
> 4. What is the Lock object? I see that some APIs require locking and some
>> don't. For example, createJob needs a Lock object as parameter, & I am
>> assuming that it is required so that one does not create multiple jobs with
>> the same JobKey.
>
> Ignore this object as it's an echo of the old client updater. It's now
> deprecated and will be removed soon. You can pass NULL for now.
>
> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>
> Another echo of the client updater but this time it's got a second life.
> Check out its new signature and comments in the api.thrift. It's
> essentially a "scale-out" API that can add instances to the existing job
> without changing the underlying task assumptions.
>
> 6. getPendingResult(TaskQuery taskquery):
>
> It's actually 'getPendingReason' and is currently used exclusively by the
> UI to get the reason for a task PENDING state.
>
> 7. setQuota & getQuota for setting user level quotas.
>
> This is to set role-level quota. Currently only required for tasks with
> 'production=True'. Search through our docs for more details.
>
> 8. killTasks to kill all running instances of a job in the cluster.
>
> It's quite versatile and can be used to kill some or all instances of the
> job.
>
> 9. startJobUpdate(JobUpdateRequest request, string message):
>
> Your observations are correct. This is the main API to change your service
> job in any way (including adding, removing or modifying instances).
>
> An aurora scheduling question is if I start a job with 5 instances, and
>> there are resources available to run only 4 of them, does the entire job
>> block, or only the 5th instance of the job blocks?
>
> Scheduler will try to schedule as many instances as it can. Those that
> will not find resources will remain in PENDING state until more resources
> are available. In your particular example only the 5th will remain PENDING.
>
>
> On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com> wrote:
>
>> Hi,
>> I was going through the Aurora Thrift API to determine how to add new
>> jobs.
>> I am using aurora v0.12 released last month and have upgraded to mesos
>> v0.25 accordingly.
>>
>> Below is a summary of my (very limited) understanding of some APIs, &
>> would help it if someone can point out flaws in my understanding:
>>
>> 1. All APIs require thrift inputs of the structs specified, and
>> return thrift values only in Response.result field.
>>
>> 2. Is there a set of examples in the documentation to help understand
>> Thrift API better?
>>
>> 3. createJob(JobDescription desc, Lock lock):
>> This is basically the API to replace the Aurora DSL/.aurora files for
>> job configuration.
>>
>> 4. What is the Lock object? I see that some APIs require locking and
>> some don't. For example, createJob needs a Lock object as parameter, & I am
>> assuming that it is required so that one does not create multiple jobs with
>> the same JobKey.
>>
>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>> By the naming convention, it seems this is used to increase the
>> number of instances of a job. It will not result in stopping of current
>> instances of the job.
>>
>> My second explanation for this API: Since it uses a set of
>> instanceIds, this is used for adding already running job in slaves to the
>> internal data structures of Aurora to track the job.
>>
>> 6. getPendingResult(TaskQuery taskquery):
>> Return the reason (in string) about why the job is PENDING. For
>> example: insufficient CPU.
>>
>> 7. setQuota & getQuota for setting user level quotas.
>>
>> 8. killTasks to kill all running instances of a job in the cluster.
>>
>> 9. startJobUpdate(JobUpdateRequest request, string message):
>> Used for updating jobs with the new TaskConfig specified. Can be used
>> if resource requirement changes. For example: If I wanted aurora to update
>> the version of container used for a job using TaskConfig.Container
>> attribute.
>>
>>
>> An aurora scheduling question is if I start a job with 5 instances, and
>> there are resources available to run only 4 of them, does the entire job
>> block, or only the 5th instance of the job blocks?
>>
>> Thanks!
>>
>> --
>> κρισhναν
>>
>
>
Re: Aurora Thrift API Info
Posted by Maxim Khutornenko <ma...@apache.org>.
>
> 1. All APIs require thrift inputs of the structs specified, and return
> thrift values only in Response.result field.
Correct. There is also 'details' field that may have additional messages
(of error or informational nature)
2. Is there a set of examples in the documentation to help understand
> Thrift API better?
The thrift API is largely undocumented. There is an effort to bring up a
fully supported REST API that will presumably get documented and become
much easier to use. It's mostly in flux now.
3. createJob(JobDescription desc, Lock lock):
This is the API to use when you a brand new service or adhoc (batch) job
created. The JobDescription is populated from the .aurora config. You may
want to trace "aurora job create" client command implementation to see how
it happens.
4. What is the Lock object? I see that some APIs require locking and some
> don't. For example, createJob needs a Lock object as parameter, & I am
> assuming that it is required so that one does not create multiple jobs with
> the same JobKey.
Ignore this object as it's an echo of the old client updater. It's now
deprecated and will be removed soon. You can pass NULL for now.
5. addInstances(AddInstancesConfig cfg, Lock lock):
Another echo of the client updater but this time it's got a second life.
Check out its new signature and comments in the api.thrift. It's
essentially a "scale-out" API that can add instances to the existing job
without changing the underlying task assumptions.
6. getPendingResult(TaskQuery taskquery):
It's actually 'getPendingReason' and is currently used exclusively by the
UI to get the reason for a task PENDING state.
7. setQuota & getQuota for setting user level quotas.
This is to set role-level quota. Currently only required for tasks with
'production=True'. Search through our docs for more details.
8. killTasks to kill all running instances of a job in the cluster.
It's quite versatile and can be used to kill some or all instances of the
job.
9. startJobUpdate(JobUpdateRequest request, string message):
Your observations are correct. This is the main API to change your service
job in any way (including adding, removing or modifying instances).
An aurora scheduling question is if I start a job with 5 instances, and
> there are resources available to run only 4 of them, does the entire job
> block, or only the 5th instance of the job blocks?
Scheduler will try to schedule as many instances as it can. Those that will
not find resources will remain in PENDING state until more resources are
available. In your particular example only the 5th will remain PENDING.
On Wed, Mar 16, 2016 at 5:54 AM, Krish <kr...@gmail.com> wrote:
> Hi,
> I was going through the Aurora Thrift API to determine how to add new jobs.
> I am using aurora v0.12 released last month and have upgraded to mesos
> v0.25 accordingly.
>
> Below is a summary of my (very limited) understanding of some APIs, &
> would help it if someone can point out flaws in my understanding:
>
> 1. All APIs require thrift inputs of the structs specified, and return
> thrift values only in Response.result field.
>
> 2. Is there a set of examples in the documentation to help understand
> Thrift API better?
>
> 3. createJob(JobDescription desc, Lock lock):
> This is basically the API to replace the Aurora DSL/.aurora files for
> job configuration.
>
> 4. What is the Lock object? I see that some APIs require locking and
> some don't. For example, createJob needs a Lock object as parameter, & I am
> assuming that it is required so that one does not create multiple jobs with
> the same JobKey.
>
> 5. addInstances(AddInstancesConfig cfg, Lock lock):
> By the naming convention, it seems this is used to increase the number
> of instances of a job. It will not result in stopping of current instances
> of the job.
>
> My second explanation for this API: Since it uses a set of
> instanceIds, this is used for adding already running job in slaves to the
> internal data structures of Aurora to track the job.
>
> 6. getPendingResult(TaskQuery taskquery):
> Return the reason (in string) about why the job is PENDING. For
> example: insufficient CPU.
>
> 7. setQuota & getQuota for setting user level quotas.
>
> 8. killTasks to kill all running instances of a job in the cluster.
>
> 9. startJobUpdate(JobUpdateRequest request, string message):
> Used for updating jobs with the new TaskConfig specified. Can be used
> if resource requirement changes. For example: If I wanted aurora to update
> the version of container used for a job using TaskConfig.Container
> attribute.
>
>
> An aurora scheduling question is if I start a job with 5 instances, and
> there are resources available to run only 4 of them, does the entire job
> block, or only the 5th instance of the job blocks?
>
> Thanks!
>
> --
> κρισhναν
>