You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nkechi Achara <nk...@googlemail.com> on 2016/01/26 22:10:03 UTC
Issues with Long subtraction in an RDD when utilising tailrecursion
down votefavorite
<http://stackoverflow.com/questions/35009560/anyone-had-issues-with-subtraction-of-a-long-within-an-rdd?noredirect=1#>
I am having an issue with the subtraction of a long within an RDD to filter
out items in the RDD that are within a certain time range.
So my code filters an RDD of case class auctions, with an object of
successfulAuctions(Long, Int, String):
auctions.filter(it => relevantAuctions(it, successfulAuctions))
The successfulAuctions object is made up of a timestamp: Long, an itemID:
Int, and a direction: String (BUY/SELL).
The relevantAuctions function basically uses tail recursion to find the
auctions in a time range for the exact item and direction.
@tailrec
def relevantAuctions(auction: Auction, successfulAuctions:
List[(Long, String, String)]): Boolean = successfulAuctions match
{
case sample :: xs => if (isRelevantAuction(auction, sample) )
true else relevantAuctions(auction, xs)
case Nil => false
}
This then feeds into another method in the if statement that checks the
timestamp in the sample is within a 10ms range, and the item ID is the
same, as is the direction.
def isRelevantAuction(auction: Auction, successfulAuction: (Long,
String, String)): Boolean = {(successfulAuction.timestampNanos -
auction.timestampNanos) >= 0 &&
(successfulAuction.timestampNanos - auction.timestampNanos) < 10000000L &&
auction.itemID == successfulAuction.itemID &&
auction.direction== successfulAuction.direction
}
I am having issues where the range option is not entirely working. The
timestamps I am receiving back are not within the required range. Although
the Item ID and direction seems to be working successfully.
The results I am getting are as follows, when I have a timestamp of
1431651108749267459 for the successful auction, I am receiving other
auctions of a time GREATER than this, where it should be less.
The auctions I am receiving have the timestamps of:
143165110874932660314316511087493307321431651108749537901
Has anyone experienced this phenomenon?
Thanks!
Fwd: Issues with Long subtraction in an RDD when utilising tailrecursion
Posted by Nkechi Achara <nk...@googlemail.com>.
Hi,
Yep, strangely I get values where the successful auction has a smaller time
than the other relevant auctions.
I have also attempted to reverse the statement, and I receive auctions that
are still greater than the successful auction. But also they are of a
greater value.
On 26 January 2016 at 22:51, Ted Yu <yu...@gmail.com> wrote:
> bq. (successfulAuction.timestampNanos - auction.timestampNanos) <
> 10000000L &&
>
> Have you included the above condition into consideration when inspecting
> timestamps of the results ?
>
> On Tue, Jan 26, 2016 at 1:10 PM, Nkechi Achara <nk...@googlemail.com>
> wrote:
>
>>
>>
>> down votefavorite
>> <http://stackoverflow.com/questions/35009560/anyone-had-issues-with-subtraction-of-a-long-within-an-rdd?noredirect=1#>
>>
>> I am having an issue with the subtraction of a long within an RDD to
>> filter out items in the RDD that are within a certain time range.
>>
>> So my code filters an RDD of case class auctions, with an object of
>> successfulAuctions(Long, Int, String):
>>
>> auctions.filter(it => relevantAuctions(it, successfulAuctions))
>>
>> The successfulAuctions object is made up of a timestamp: Long, an itemID:
>> Int, and a direction: String (BUY/SELL).
>>
>> The relevantAuctions function basically uses tail recursion to find the
>> auctions in a time range for the exact item and direction.
>>
>> @tailrec
>> def relevantAuctions(auction: Auction, successfulAuctions: List[(Long, String, String)]): Boolean = successfulAuctions match {
>> case sample :: xs => if (isRelevantAuction(auction, sample) ) true else relevantAuctions(auction, xs)
>> case Nil => false
>> }
>>
>> This then feeds into another method in the if statement that checks the
>> timestamp in the sample is within a 10ms range, and the item ID is the
>> same, as is the direction.
>>
>> def isRelevantAuction(auction: Auction, successfulAuction: (Long, String, String)): Boolean = {(successfulAuction.timestampNanos - auction.timestampNanos) >= 0 &&
>> (successfulAuction.timestampNanos - auction.timestampNanos) < 10000000L &&
>> auction.itemID == successfulAuction.itemID &&
>> auction.direction== successfulAuction.direction
>> }
>>
>> I am having issues where the range option is not entirely working. The
>> timestamps I am receiving back are not within the required range. Although
>> the Item ID and direction seems to be working successfully.
>>
>> The results I am getting are as follows, when I have a timestamp of
>> 1431651108749267459 for the successful auction, I am receiving other
>> auctions of a time GREATER than this, where it should be less.
>>
>> The auctions I am receiving have the timestamps of:
>>
>> 143165110874932660314316511087493307321431651108749537901
>>
>> Has anyone experienced this phenomenon?
>>
>> Thanks!
>>
>
>
Re: Issues with Long subtraction in an RDD when utilising tailrecursion
Posted by Ted Yu <yu...@gmail.com>.
bq. (successfulAuction.timestampNanos - auction.timestampNanos) < 10000000L
&&
Have you included the above condition into consideration when inspecting
timestamps of the results ?
On Tue, Jan 26, 2016 at 1:10 PM, Nkechi Achara <nk...@googlemail.com>
wrote:
>
>
> down votefavorite
> <http://stackoverflow.com/questions/35009560/anyone-had-issues-with-subtraction-of-a-long-within-an-rdd?noredirect=1#>
>
> I am having an issue with the subtraction of a long within an RDD to
> filter out items in the RDD that are within a certain time range.
>
> So my code filters an RDD of case class auctions, with an object of
> successfulAuctions(Long, Int, String):
>
> auctions.filter(it => relevantAuctions(it, successfulAuctions))
>
> The successfulAuctions object is made up of a timestamp: Long, an itemID:
> Int, and a direction: String (BUY/SELL).
>
> The relevantAuctions function basically uses tail recursion to find the
> auctions in a time range for the exact item and direction.
>
> @tailrec
> def relevantAuctions(auction: Auction, successfulAuctions: List[(Long, String, String)]): Boolean = successfulAuctions match {
> case sample :: xs => if (isRelevantAuction(auction, sample) ) true else relevantAuctions(auction, xs)
> case Nil => false
> }
>
> This then feeds into another method in the if statement that checks the
> timestamp in the sample is within a 10ms range, and the item ID is the
> same, as is the direction.
>
> def isRelevantAuction(auction: Auction, successfulAuction: (Long, String, String)): Boolean = {(successfulAuction.timestampNanos - auction.timestampNanos) >= 0 &&
> (successfulAuction.timestampNanos - auction.timestampNanos) < 10000000L &&
> auction.itemID == successfulAuction.itemID &&
> auction.direction== successfulAuction.direction
> }
>
> I am having issues where the range option is not entirely working. The
> timestamps I am receiving back are not within the required range. Although
> the Item ID and direction seems to be working successfully.
>
> The results I am getting are as follows, when I have a timestamp of
> 1431651108749267459 for the successful auction, I am receiving other
> auctions of a time GREATER than this, where it should be less.
>
> The auctions I am receiving have the timestamps of:
>
> 143165110874932660314316511087493307321431651108749537901
>
> Has anyone experienced this phenomenon?
>
> Thanks!
>