You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by vr spark <vr...@gmail.com> on 2016/08/17 15:46:30 UTC
Attempting to accept an unknown offer
W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492
W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493
W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494
W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495
W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496
W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497
W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498
W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499
W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500
2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@
zookeeper_interest@1557: Exceeded deadline by 13ms
2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@
zookeeper_interest@1557: Exceeded deadline by 11ms
Re: Attempting to accept an unknown offer
Posted by Michael Gummelt <mg...@mesosphere.io>.
That error message occurs when the Mesos scheduler tries to accept an offer
that doesn't exist. It should never happen. Can you submit a JIRA and cc
me to it? Also, what libmesos and mesos master version are you running?
On Wed, Aug 17, 2016 at 9:23 AM, vr spark <vr...@gmail.com> wrote:
> My code is very simple, if i use other hive tables, my code works fine. This
> particular table (virtual view) is huge and might have more metadata.
>
> It has only two columns.
>
> virtual view name is : cluster_table
>
> # col_name data_type
>
> ln string
>
> parti int
>
>
> here is snippet...
>
> from pyspark import SparkConf, SparkContext
>
> from pyspark.sql import HiveContext
>
> import pyspark.sql
>
> import json
>
> myconf=SparkConf().setAppName("sql")
>
> spcont=SparkContext(conf=myconf)
>
> sqlcont=HiveContext(spcont)
>
> res=sqlcont.sql("select parti FROM h.cluster_table WHERE parti > 408910
> and parti <408911 limit 10")
>
> print res.printSchema()
>
> print 'res'
>
> print res
>
> df=res.collect()
>
> print 'after collect'
>
> print df
>
>
> Here is the ouput after i submit the job
>
> I0817 09:18:40.606465 31409 sched.cpp:262] New master detected at
> master@x.y.17.56:6750
>
> I0817 09:18:40.607461 31409 sched.cpp:272] No credentials provided.
> Attempting to register without authentication
>
> I0817 09:18:40.612763 31409 sched.cpp:641] Framework registered with
> b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6336
>
> 16/08/17 09:18:57 WARN ObjectStore: Version information not found in
> metastore. hive.metastore.schema.verification is not enabled so recording
> the schema version 1.2.0
>
> 16/08/17 09:18:57 WARN ObjectStore: Failed to get database default,
> returning NoSuchObjectException
>
> root
>
> |-- parti: integer (nullable = true)
>
>
> None
>
> res
>
> DataFrame[partition_epoch_hourtenth: int]
>
> 2016-08-17 09:19:20,648:31315(0x7fafebfb1700):ZOO_WARN@
> zookeeper_interest@1557: Exceeded deadline by 19ms
>
> 2016-08-17 09:19:30,662:31315(0x7fafebfb1700):ZOO_WARN@
> zookeeper_interest@1557: Exceeded deadline by 13ms
>
> W0817 09:20:01.715824 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676564
>
> W0817 09:20:01.716455 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676630
>
> W0817 09:20:01.716645 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676713
>
> W0817 09:20:01.724409 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676554
>
> W0817 09:20:01.724728 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676555
>
> W0817 09:20:01.724936 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676556
>
> W0817 09:20:01.725126 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676557
>
> W0817 09:20:01.725309 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676558.
>
> and many more lines like this on the screen with similar message....
>
>
>
> On Wed, Aug 17, 2016 at 9:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Please include user@ in your reply.
>>
>> Can you reveal the snippet of hive sql ?
>>
>> On Wed, Aug 17, 2016 at 9:04 AM, vr spark <vr...@gmail.com> wrote:
>>
>>> spark 1.6.1
>>> mesos
>>> job is running for like 10-15 minutes and giving this message and i
>>> killed it.
>>>
>>> In this job, i am creating data frame from a hive sql. There are other
>>> similar jobs which work fine
>>>
>>> On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Can you provide more information ?
>>>>
>>>> Were you running on YARN ?
>>>> Which version of Spark are you using ?
>>>>
>>>> Was your job failing ?
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Aug 17, 2016 at 8:46 AM, vr spark <vr...@gmail.com> wrote:
>>>>
>>>>>
>>>>> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492
>>>>>
>>>>> W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493
>>>>>
>>>>> W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494
>>>>>
>>>>> W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495
>>>>>
>>>>> W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496
>>>>>
>>>>> W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497
>>>>>
>>>>> W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498
>>>>>
>>>>> W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499
>>>>>
>>>>> W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500
>>>>>
>>>>> 2016-08-16 23:18:41,877:16226(0x7f71271b6
>>>>> 700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 13ms
>>>>>
>>>>> 2016-08-16 23:21:12,007:16226(0x7f71271b6
>>>>> 700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 11ms
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
--
Michael Gummelt
Software Engineer
Mesosphere
Re: Attempting to accept an unknown offer
Posted by vr spark <vr...@gmail.com>.
My code is very simple, if i use other hive tables, my code works fine. This
particular table (virtual view) is huge and might have more metadata.
It has only two columns.
virtual view name is : cluster_table
# col_name data_type
ln string
parti int
here is snippet...
from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext
import pyspark.sql
import json
myconf=SparkConf().setAppName("sql")
spcont=SparkContext(conf=myconf)
sqlcont=HiveContext(spcont)
res=sqlcont.sql("select parti FROM h.cluster_table WHERE parti > 408910
and parti <408911 limit 10")
print res.printSchema()
print 'res'
print res
df=res.collect()
print 'after collect'
print df
Here is the ouput after i submit the job
I0817 09:18:40.606465 31409 sched.cpp:262] New master detected at
master@x.y.17.56:6750
I0817 09:18:40.607461 31409 sched.cpp:272] No credentials provided.
Attempting to register without authentication
I0817 09:18:40.612763 31409 sched.cpp:641] Framework registered with
b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6336
16/08/17 09:18:57 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording
the schema version 1.2.0
16/08/17 09:18:57 WARN ObjectStore: Failed to get database default,
returning NoSuchObjectException
root
|-- parti: integer (nullable = true)
None
res
DataFrame[partition_epoch_hourtenth: int]
2016-08-17 09:19:20,648:31315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 19ms
2016-08-17 09:19:30,662:31315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 13ms
W0817 09:20:01.715824 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676564
W0817 09:20:01.716455 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676630
W0817 09:20:01.716645 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676713
W0817 09:20:01.724409 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676554
W0817 09:20:01.724728 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676555
W0817 09:20:01.724936 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676556
W0817 09:20:01.725126 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676557
W0817 09:20:01.725309 31412 sched.cpp:1195] Attempting to accept an unknown
offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676558.
and many more lines like this on the screen with similar message....
On Wed, Aug 17, 2016 at 9:08 AM, Ted Yu <yu...@gmail.com> wrote:
> Please include user@ in your reply.
>
> Can you reveal the snippet of hive sql ?
>
> On Wed, Aug 17, 2016 at 9:04 AM, vr spark <vr...@gmail.com> wrote:
>
>> spark 1.6.1
>> mesos
>> job is running for like 10-15 minutes and giving this message and i
>> killed it.
>>
>> In this job, i am creating data frame from a hive sql. There are other
>> similar jobs which work fine
>>
>> On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you provide more information ?
>>>
>>> Were you running on YARN ?
>>> Which version of Spark are you using ?
>>>
>>> Was your job failing ?
>>>
>>> Thanks
>>>
>>> On Wed, Aug 17, 2016 at 8:46 AM, vr spark <vr...@gmail.com> wrote:
>>>
>>>>
>>>> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492
>>>>
>>>> W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493
>>>>
>>>> W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494
>>>>
>>>> W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495
>>>>
>>>> W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496
>>>>
>>>> W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497
>>>>
>>>> W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498
>>>>
>>>> W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499
>>>>
>>>> W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an
>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500
>>>>
>>>> 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere
>>>> st@1557: Exceeded deadline by 13ms
>>>>
>>>> 2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere
>>>> st@1557: Exceeded deadline by 11ms
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Attempting to accept an unknown offer
Posted by Ted Yu <yu...@gmail.com>.
Please include user@ in your reply.
Can you reveal the snippet of hive sql ?
On Wed, Aug 17, 2016 at 9:04 AM, vr spark <vr...@gmail.com> wrote:
> spark 1.6.1
> mesos
> job is running for like 10-15 minutes and giving this message and i killed
> it.
>
> In this job, i am creating data frame from a hive sql. There are other
> similar jobs which work fine
>
> On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you provide more information ?
>>
>> Were you running on YARN ?
>> Which version of Spark are you using ?
>>
>> Was your job failing ?
>>
>> Thanks
>>
>> On Wed, Aug 17, 2016 at 8:46 AM, vr spark <vr...@gmail.com> wrote:
>>
>>>
>>> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492
>>>
>>> W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493
>>>
>>> W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494
>>>
>>> W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495
>>>
>>> W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496
>>>
>>> W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497
>>>
>>> W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498
>>>
>>> W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499
>>>
>>> W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an
>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500
>>>
>>> 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere
>>> st@1557: Exceeded deadline by 13ms
>>>
>>> 2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere
>>> st@1557: Exceeded deadline by 11ms
>>>
>>>
>>>
>>>
>>
>
Re: Attempting to accept an unknown offer
Posted by Ted Yu <yu...@gmail.com>.
Can you provide more information ?
Were you running on YARN ?
Which version of Spark are you using ?
Was your job failing ?
Thanks
On Wed, Aug 17, 2016 at 8:46 AM, vr spark <vr...@gmail.com> wrote:
>
> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492
>
> W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493
>
> W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494
>
> W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495
>
> W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496
>
> W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497
>
> W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498
>
> W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499
>
> W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500
>
> 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper_
> interest@1557: Exceeded deadline by 13ms
>
> 2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@zookeeper_
> interest@1557: Exceeded deadline by 11ms
>
>
>
>