user@spark.apache.org, 2016-12

You are viewing a plain text version of this content. The canonical link for it is here.

- Spark Standalone Cluster - Running applications in JSON format - posted by Carl Ballantyne <ca...@guvera.com> on 2016/12/01 05:20:02 UTC, 6 replies.
- SPARK-SUBMIT and optional args like -h etc - posted by "Patnaik, Vandana" <Va...@infogroup.com> on 2016/12/01 05:23:24 UTC, 0 replies.
- Re: updateStateByKey -- when the key is multi-column (like a composite key ) - posted by shyla deshpande <de...@gmail.com> on 2016/12/01 06:20:56 UTC, 0 replies.
- mapWithState's checkpoint size - posted by Bhavani Sudha Saktheeswaran <bh...@uber.com> on 2016/12/01 06:25:24 UTC, 1 replies.
- Re: java.lang.Exception: Could not compute split, block input-0-1480539568000 not found - posted by kant kodali <ka...@gmail.com> on 2016/12/01 06:40:19 UTC, 5 replies.
- Spark 2.0.2 , using DStreams in Spark Streaming . How do I create SQLContext? Please help - posted by shyla deshpande <de...@gmail.com> on 2016/12/01 06:57:58 UTC, 2 replies.
- Re: SPARK-SUBMIT and optional args like -h etc - posted by Daniel van der Ende <da...@gmail.com> on 2016/12/01 06:58:39 UTC, 0 replies.
- Re: Spark Job not exited and shows running - posted by Daniel van der Ende <da...@gmail.com> on 2016/12/01 07:03:04 UTC, 2 replies.
- RE: PySpark to remote cluster - posted by "Schaefers, Klaus" <kl...@philips.com> on 2016/12/01 08:38:55 UTC, 0 replies.
- Re: [structured streaming] How to remove outdated data when use Window Operations - posted by Tathagata Das <ta...@gmail.com> on 2016/12/01 08:53:34 UTC, 3 replies.
- How to Check Dstream is empty or not? - posted by "rockinfeet@gmail.com" <ro...@gmail.com> on 2016/12/01 12:28:17 UTC, 1 replies.
- newly added Executors couldn't fetch jar files from Master - posted by Evgenii Morozov <ev...@gmail.com> on 2016/12/01 14:28:58 UTC, 0 replies.
- Spark-shell doesn't see changes coming from Kafka topic - posted by Otávio Carvalho <ot...@gmail.com> on 2016/12/01 14:31:12 UTC, 2 replies.
- RE: build models in parallel - posted by Masood Krohy <ma...@intact.net> on 2016/12/01 15:23:43 UTC, 0 replies.
- Unsubscribe - posted by hardik nagda <ha...@gmail.com> on 2016/12/01 16:03:27 UTC, 27 replies.
- Spark 2.x Pyspark Spark SQL createDataframe Error - posted by Vinayak Joshi5 <vi...@in.ibm.com> on 2016/12/01 17:23:10 UTC, 0 replies.
- Re: Spark 2.x Pyspark Spark SQL createDataframe Error - posted by Vinayak Joshi5 <vi...@in.ibm.com> on 2016/12/01 17:33:22 UTC, 2 replies.
- support vector regression in spark - posted by roni <ro...@gmail.com> on 2016/12/01 20:08:18 UTC, 0 replies.
- unsubscribe - posted by Vishal Soni <vs...@snaplogic.com> on 2016/12/01 20:14:27 UTC, 17 replies.
- quick question - posted by kant kodali <ka...@gmail.com> on 2016/12/01 20:25:07 UTC, 0 replies.
- [GraphFrame, Pyspark] Weighted Edge in PageRank - posted by Weiwei Zhang <wz...@dons.usfca.edu> on 2016/12/01 22:41:30 UTC, 2 replies.
- [Spark Dataset]: How to conduct co-partition join in the new Dataset API in Spark 2.0 - posted by Dale Wang <w....@gmail.com> on 2016/12/02 05:23:31 UTC, 1 replies.
- Usage of -javaagent with spark.executor.extrajavaoptions configuration - posted by Kanchan W <w....@gmail.com> on 2016/12/02 05:41:41 UTC, 0 replies.
- Accessing log for lost executors - posted by Nisrina Luthfiyati <ni...@gmail.com> on 2016/12/02 06:30:46 UTC, 1 replies.
- Spark sql generated dynamically - posted by Georg Heiler <ge...@gmail.com> on 2016/12/02 11:33:03 UTC, 2 replies.
- Kafka 0.10 & Spark Streaming 2.0.2 - posted by gabrielperez2484 <ga...@adtheorent.com> on 2016/12/02 13:43:58 UTC, 7 replies.
- RDD flatmap to multiple key/value pairs - posted by im281 <im...@gmail.com> on 2016/12/02 16:05:40 UTC, 1 replies.
- Re: TallSkinnyQR - posted by Iman Mohtashemi <im...@gmail.com> on 2016/12/02 16:13:02 UTC, 9 replies.
- RDD getPartitions() size and HashPartitioner numPartitions - posted by Amit Sela <am...@gmail.com> on 2016/12/02 21:53:12 UTC, 1 replies.
- What benefits do we really get out of colocation? - posted by kant kodali <ka...@gmail.com> on 2016/12/03 06:38:26 UTC, 12 replies.
- Parquet timestamp storage in Hive and possible use case of spark instead of impala - posted by Mich Talebzadeh <mi...@gmail.com> on 2016/12/03 12:11:56 UTC, 0 replies.
- Design patterns for Spark implementation - posted by Vasu Gourabathina <vg...@gmail.com> on 2016/12/03 16:59:58 UTC, 8 replies.
- Spark shuffle: FileNotFound exception - posted by Swapnil Shinde <sw...@gmail.com> on 2016/12/04 06:03:04 UTC, 1 replies.
- Creating schema from json representation - posted by "Mendelson, Assaf" <As...@rsa.com> on 2016/12/04 09:10:37 UTC, 1 replies.
- Access multiple cluster - posted by ayan guha <gu...@gmail.com> on 2016/12/04 21:45:28 UTC, 4 replies.
- Re: SVM regression in Spark - posted by Evgenii Morozov <ev...@gmail.com> on 2016/12/04 22:55:12 UTC, 0 replies.
- Can spark support exactly once based kafka ? Due to these following question? - posted by John Fang <xi...@alibaba-inc.com> on 2016/12/04 23:49:15 UTC, 4 replies.
- how to use Dataset of transform method - posted by LQ <20...@qq.com> on 2016/12/05 01:33:25 UTC, 0 replies.
- How to convert a unix timestamp column into date format(yyyy-MM-dd) ? - posted by "Devi P.V" <de...@gmail.com> on 2016/12/05 07:38:00 UTC, 7 replies.
- Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL - posted by kant kodali <ka...@gmail.com> on 2016/12/05 10:51:31 UTC, 3 replies.
- Kerberos and YARN - functions in spark-shell and spark submit local but not cluster mode - posted by Gerard Casey <ge...@gmail.com> on 2016/12/05 12:35:44 UTC, 18 replies.
- Livy with Spark - posted by Mich Talebzadeh <mi...@gmail.com> on 2016/12/05 14:33:12 UTC, 5 replies.
- Spark streaming completed batches statistics - posted by Richard Startin <ri...@outlook.com> on 2016/12/05 15:55:36 UTC, 4 replies.
- How could one specify a Docker image for each job to be used by executors? - posted by Enno Shioji <es...@gmail.com> on 2016/12/05 16:54:47 UTC, 0 replies.
- streaming deployment on yarn -emr - posted by "Saurabh Malviya (samalviy)" <sa...@cisco.com> on 2016/12/05 17:12:07 UTC, 0 replies.
- Re: Would spark dataframe/rdd read from external source on every action? - posted by neil90 <ne...@icloud.com> on 2016/12/05 17:23:58 UTC, 0 replies.
- Writing DataFrame filter results to separate files - posted by Everett Anderson <ev...@nuna.com.INVALID> on 2016/12/05 18:59:36 UTC, 5 replies.
- Collaborative Filtering Implicit Feedback Impl. - posted by Jerry Lam <ch...@gmail.com> on 2016/12/05 19:42:21 UTC, 5 replies.
- Pretrained Word2Vec models - posted by Lee Becker <le...@hapara.com> on 2016/12/05 21:34:13 UTC, 1 replies.
- Back-pressure to Spark Kafka Streaming? - posted by Liren Ding <sk...@gmail.com> on 2016/12/05 22:18:01 UTC, 2 replies.
- Streaming audio files - posted by habibbaluwala <ha...@gmail.com> on 2016/12/05 22:27:36 UTC, 0 replies.
- Monitoring the User Metrics for a long running Spark Job - posted by "Chawla,Sumit " <su...@gmail.com> on 2016/12/06 00:17:48 UTC, 7 replies.
- Please unsubscribe me - posted by Srinivas Potluri <sh...@gmail.com> on 2016/12/06 00:19:33 UTC, 0 replies.
- custom generate spark application id - posted by rtijoriwala <ti...@gmail.com> on 2016/12/06 00:35:34 UTC, 1 replies.
- [Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ? - posted by sancheng <sa...@gmail.com> on 2016/12/06 01:35:19 UTC, 2 replies.
- driver in queued state and not started - posted by Yu Wei <yu...@hotmail.com> on 2016/12/06 06:05:06 UTC, 1 replies.
- doing streaming efficiently - posted by "Mendelson, Assaf" <As...@rsa.com> on 2016/12/06 08:39:48 UTC, 0 replies.
- how to add colum to dataframe - posted by lk_spark <lk...@163.com> on 2016/12/06 09:35:51 UTC, 3 replies.
- How to compute the recall and F1-score in Linear Regression based model - posted by "Md. Rezaul Karim" <re...@insight-centre.org> on 2016/12/06 11:14:04 UTC, 3 replies.
- [GraphX] Extreme scheduler delay - posted by AntonIpp <an...@simudyne.com> on 2016/12/06 11:49:52 UTC, 1 replies.
- get corrupted rows using columnNameOfCorruptRecord - posted by Yehuda Finkelstein <ye...@veracity-group.com> on 2016/12/06 14:31:51 UTC, 4 replies.
- Re: Spark Streaming - join streaming and static data - posted by Burak Yavuz <br...@gmail.com> on 2016/12/06 17:11:00 UTC, 1 replies.
- efficient filtering on a dataframe - posted by Koert Kuipers <ko...@tresata.com> on 2016/12/06 19:04:44 UTC, 0 replies.
- MLlib to Compute boundaries of a rectangle given random points on its Surface - posted by Pradeep Gaddam <Pr...@viewglass.com> on 2016/12/06 22:51:53 UTC, 0 replies.
- - posted by ayan guha <gu...@gmail.com> on 2016/12/06 23:20:34 UTC, 1 replies.
- [Spark SQL]: Dataset Encoder equivalent for pre 1.6.0 releases? - posted by Denis Papathanasiou <de...@gmail.com> on 2016/12/06 23:43:18 UTC, 0 replies.
- credentials are not hiding on a jdbc query - posted by Cesar <ce...@gmail.com> on 2016/12/07 00:33:52 UTC, 0 replies.
- OutOfMemoryError while running job... - posted by Kevin Burton <bu...@spinn3r.com> on 2016/12/07 00:36:13 UTC, 0 replies.
- Terminate job without killing - posted by Bruno Faria <br...@hotmail.com> on 2016/12/07 03:03:16 UTC, 1 replies.
- Dynamically applying schema in spark. - posted by Satwinder Singh <sa...@gmail.com> on 2016/12/07 03:40:23 UTC, 0 replies.
- Adding a new nested column into dataframe - posted by AS...@nz.imshealth.com on 2016/12/07 05:09:26 UTC, 0 replies.
- Not per-key state - posted by Anty Rao <an...@gmail.com> on 2016/12/07 08:13:15 UTC, 0 replies.
- Identifying DataFrames in executing tasks - posted by Aniket More <An...@bitwiseglobal.com> on 2016/12/07 09:27:59 UTC, 0 replies.
- Spark 2.0.2 with Spark JobServer - posted by Jose Carlos Guevara Turruelles <jo...@value-iq.com> on 2016/12/07 09:58:27 UTC, 0 replies.
- Debugging persistence of custom estimator - posted by geoHeil <ge...@gmail.com> on 2016/12/07 10:03:59 UTC, 0 replies.
- Not per-key state in spark streaming - posted by Anty Rao <an...@gmail.com> on 2016/12/07 11:42:58 UTC, 5 replies.
- filter RDD by variable - posted by "Soheila S." <so...@gmail.com> on 2016/12/07 16:22:32 UTC, 1 replies.
- create new spark context from ipython or jupyter - posted by pseudo oduesp <ps...@gmail.com> on 2016/12/07 17:18:42 UTC, 0 replies.
- Reprocessing failed jobs in Streaming job - posted by map reduced <k3...@gmail.com> on 2016/12/07 19:46:19 UTC, 6 replies.
- Driver/Executor Memory values during Unit Testing - posted by Aleksander Eskilson <al...@gmail.com> on 2016/12/07 21:44:19 UTC, 0 replies.
- Accessing classpath resources in Spark Shell - posted by Michal Šenkýř <mi...@gmail.com> on 2016/12/07 21:48:49 UTC, 0 replies.
- Pruning decision tree to create an optimal tree - posted by "Md. Rezaul Karim" <re...@insight-centre.org> on 2016/12/07 22:10:22 UTC, 0 replies.
- Running spark from Eclipse and then Jar - posted by im281 <im...@gmail.com> on 2016/12/07 23:18:12 UTC, 13 replies.
- StreamingContext.textFileStream(...) - posted by muthu <ba...@gmail.com> on 2016/12/08 00:33:03 UTC, 0 replies.
- WARN util.NativeCodeLoader - posted by baipeng <bp...@meitu.com> on 2016/12/08 06:38:25 UTC, 2 replies.
- How to clean the cache when i do performance test in spark - posted by "Zhang, Liyun" <li...@intel.com> on 2016/12/08 07:47:03 UTC, 0 replies.
- How to find unique values after groupBy() in spark dataframe ? - posted by "Devi P.V" <de...@gmail.com> on 2016/12/08 08:38:02 UTC, 1 replies.
- Re: Publishing of the Spectral LDA model on Spark Packages - posted by François Garillot <fr...@garillot.net> on 2016/12/08 10:32:14 UTC, 0 replies.
- "Failed to find data source: libsvm" while running Spark application with jar - posted by "Md. Rezaul Karim" <re...@insight-centre.org> on 2016/12/08 10:52:09 UTC, 0 replies.
- Managed memory leak : spark-2.0.2 - posted by Appu K <ku...@gmail.com> on 2016/12/08 11:10:43 UTC, 2 replies.
- few basic questions on structured streaming - posted by kant kodali <ka...@gmail.com> on 2016/12/08 11:50:25 UTC, 2 replies.
- Records processed metric for intermediate datasets - posted by Aniket R More <An...@bitwiseglobal.com> on 2016/12/08 13:18:56 UTC, 0 replies.
- TO ALL WHO WANT TO UNSUBSCRIBE - posted by 5g...@guerrillamail.com on 2016/12/08 14:49:05 UTC, 0 replies.
- Re: unscribe - posted by Nicholas Chammas <ni...@gmail.com> on 2016/12/08 14:57:17 UTC, 0 replies.
- unit testing in spark - posted by pseudo oduesp <ps...@gmail.com> on 2016/12/08 15:14:05 UTC, 9 replies.
- Re: OS killing Executor due to high (possibly off heap) memory usage - posted by Aniket Bhatnagar <an...@gmail.com> on 2016/12/08 16:53:10 UTC, 0 replies.
- spark reshape hive table and save to parquet - posted by Anton Kravchenko <kr...@gmail.com> on 2016/12/08 16:53:25 UTC, 4 replies.
- Question about the DirectKafkaInputDStream - posted by John Fang <xi...@alibaba-inc.com> on 2016/12/08 17:50:55 UTC, 0 replies.
- When will Structured Streaming support stream-to-stream joins? - posted by ljwagerfield <la...@dmz.wagerfield.com> on 2016/12/08 19:16:06 UTC, 2 replies.
- .tar.bz2 in spark - posted by Maurin Lenglart <ma...@cuberonlabs.com> on 2016/12/08 19:18:11 UTC, 1 replies.
- SparkContext not creating due Logger initialization - posted by Adnan Ahmed <le...@hotmail.com> on 2016/12/08 19:29:00 UTC, 0 replies.
- Re: Spark app write too many small parquet files - posted by Kevin Tran <ke...@gmail.com> on 2016/12/08 21:03:13 UTC, 1 replies.
- Phoenix Plugin for Spark - connecting to Phoenix in secured cluster. - posted by Marcin Pastecki <ma...@gmail.com> on 2016/12/08 21:21:03 UTC, 0 replies.
- KMediods in Spark java - posted by Shak S <sh...@gmail.com> on 2016/12/08 21:39:50 UTC, 0 replies.
- Fwd: Question about SPARK-11374 (skip.header.line.count) - posted by Dongjoon Hyun <do...@apache.org> on 2016/12/09 00:01:11 UTC, 0 replies.
- Issue in using DenseVector in RowMatrix, error could be due to ml and mllib package changes - posted by satyajit vegesna <sa...@gmail.com> on 2016/12/09 02:42:35 UTC, 1 replies.
- flatmap pair - posted by im281 <im...@gmail.com> on 2016/12/09 03:11:22 UTC, 0 replies.
- how can I set the log configuration file for spark history server ? - posted by John Fang <xi...@alibaba-inc.com> on 2016/12/09 03:20:45 UTC, 0 replies.
- Re: how can I set the log configuration file for spark history server ? - posted by Don Drake <do...@gmail.com> on 2016/12/09 04:03:08 UTC, 1 replies.
- reading data from s3 - posted by Hitesh Goyal <hi...@nlpcaptcha.com> on 2016/12/09 05:42:37 UTC, 1 replies.
- Few questions on reliability of accumulators value. - posted by Sudev A C <su...@goibibo.com> on 2016/12/09 08:58:18 UTC, 4 replies.
- About transformations - posted by brccosta <br...@gmail.com> on 2016/12/09 11:24:06 UTC, 2 replies.
- groupByKey vs reduceByKey - posted by Appu K <ku...@gmail.com> on 2016/12/09 12:18:27 UTC, 0 replies.
- problem with kafka createDirectStream .. - posted by Debasish Ghosh <gh...@gmail.com> on 2016/12/09 16:14:34 UTC, 10 replies.
- Information required - posted by Rishabh Wadhawan <ri...@gmail.com> on 2016/12/09 17:25:49 UTC, 0 replies.
- SparkSQL - posted by Niraj Kumar <nk...@incedoinc.com> on 2016/12/09 18:47:49 UTC, 0 replies.
- Document Similarity -Spark Mllib - posted by satyajit vegesna <sa...@gmail.com> on 2016/12/09 19:52:42 UTC, 0 replies.
- Wrting data from Spark streaming to AWS Redshift? - posted by shyla deshpande <de...@gmail.com> on 2016/12/09 21:25:06 UTC, 2 replies.
- Spark job server pros and cons - posted by Cassa L <lc...@gmail.com> on 2016/12/09 22:31:48 UTC, 1 replies.
- Random Forest hangs without trace of error - posted by mhornbech <mo...@datasolvr.com> on 2016/12/09 23:28:46 UTC, 5 replies.
- [Spark log4j] Turning off log4j while scala program runs on spark-submit - posted by Irving Duran <ir...@gmail.com> on 2016/12/10 02:40:09 UTC, 1 replies.
- Spark Streaming with Kafka - posted by Anton Okolnychyi <an...@gmail.com> on 2016/12/11 14:51:46 UTC, 5 replies.
- Spark 2 or Spark 1.6.x? - posted by Lohith Samaga M <Lo...@mphasis.com> on 2016/12/12 06:34:10 UTC, 2 replies.
- using replace function - posted by Hitesh Goyal <hi...@nlpcaptcha.com> on 2016/12/12 07:54:35 UTC, 1 replies.
- Graphx triplet loops causing null pointer exception - posted by balaji9058 <ks...@gmail.com> on 2016/12/12 10:47:39 UTC, 0 replies.
- [Spark Core] - Spark dynamoDB integration - posted by Pratyaksh Sharma <pr...@itbhu.ac.in> on 2016/12/12 11:03:04 UTC, 2 replies.
- Livy VS Spark Job Server - posted by shyla deshpande <de...@gmail.com> on 2016/12/12 17:50:27 UTC, 0 replies.
- Tensor Flow - posted by Meeraj Kunnumpurath <me...@servicesymphony.com> on 2016/12/12 18:23:32 UTC, 1 replies.
- Optimization for Processing a million of HTML files - posted by Reth RM <re...@gmail.com> on 2016/12/12 23:06:46 UTC, 2 replies.
- wholeTextFiles() - posted by Pradeep <pr...@mail.com> on 2016/12/13 03:34:01 UTC, 2 replies.
- Query in SparkSQL - posted by Niraj Kumar <nk...@incedoinc.com> on 2016/12/13 04:21:04 UTC, 1 replies.
- Where is yarn-shuffle.jar in maven? - posted by Neal Yin <ne...@workday.com> on 2016/12/13 05:56:41 UTC, 2 replies.
- Dynamic spark sql - posted by geoHeil <ge...@gmail.com> on 2016/12/13 06:23:39 UTC, 0 replies.
- Graphx triplet comparison - posted by balaji9058 <ks...@gmail.com> on 2016/12/13 06:52:02 UTC, 3 replies.
- sbt "org.apache.spark#spark-streaming-kafka_2.11;2.0.0: not found" - posted by Luke Adolph <ke...@gmail.com> on 2016/12/13 07:30:22 UTC, 1 replies.
- How to set classpath for a job that submit to Mesos cluster - posted by Chanh Le <gi...@gmail.com> on 2016/12/13 10:20:56 UTC, 0 replies.
- About Spark Multiple Shared Context with Spark 2.0 - posted by Chetan Khatri <ck...@gmail.com> on 2016/12/13 11:42:03 UTC, 1 replies.
- Output Side Effects for different chain of operations - posted by "Chawla,Sumit " <su...@gmail.com> on 2016/12/13 16:31:16 UTC, 3 replies.
- Streaming Batch Oddities - posted by Bryan Jeffrey <br...@gmail.com> on 2016/12/13 17:11:27 UTC, 1 replies.
- Fwd: [Spark-SQL] collect_list() support for nested collection - posted by Ninad Shringarpure <ni...@cloudera.com> on 2016/12/13 18:43:17 UTC, 2 replies.
- Re: Third party library - posted by vineet chadha <st...@gmail.com> on 2016/12/13 19:02:44 UTC, 2 replies.
- Parallel read from OracleDB slow, fails on large tables - posted by epettijohn <vi...@gmail.com> on 2016/12/14 00:54:31 UTC, 0 replies.
- Belief propagation algorithm is open sourced - posted by "Ulanov, Alexander" <al...@hpe.com> on 2016/12/14 00:57:52 UTC, 3 replies.
- Does Spark 2.0.2 sql syntax not support hive sql syntax any more? - posted by qmzhang <57...@qq.com> on 2016/12/14 09:19:28 UTC, 0 replies.
- can we unite the UI among different standaone clusters' UI? - posted by John Fang <xi...@alibaba-inc.com> on 2016/12/14 13:14:52 UTC, 0 replies.
- coalesce ending up very unbalanced - but why? - posted by Adrian Bridgett <ad...@opensignal.com> on 2016/12/14 13:56:24 UTC, 5 replies.
- Proper use of map(..., Encoder) - posted by Brad Cox <br...@gmail.com> on 2016/12/14 16:05:13 UTC, 0 replies.
- Hortonworks Spark Certification Query - posted by Aakash Basu <aa...@gmail.com> on 2016/12/14 17:02:48 UTC, 0 replies.
- Handling Exception or Control in spark dataframe write() - posted by bhayat <ba...@gmail.com> on 2016/12/14 18:10:43 UTC, 1 replies.
- TaskSetManager stalls for 1 min in the middle of a job - posted by Oleg Mazurov <om...@splicemachine.com> on 2016/12/14 22:53:23 UTC, 0 replies.
- How many Spark streaming applications can be run at a time on a Spark cluster? - posted by shyla deshpande <de...@gmail.com> on 2016/12/15 00:42:10 UTC, 4 replies.
- How to get recent value in spark dataframe - posted by Milin korath <mi...@impelsys.com> on 2016/12/15 04:49:22 UTC, 8 replies.
- Mlib: TF-IDF Computation Improvement - posted by Reth RM <re...@gmail.com> on 2016/12/15 07:07:46 UTC, 0 replies.
- "remember" vs "window" in Spark Streaming - posted by Mattz <ea...@gmail.com> on 2016/12/15 11:20:30 UTC, 0 replies.
- spark use /tmp directory instead of directory from spark.local.dir - posted by AlexModestov <Al...@gmail.com> on 2016/12/15 13:13:04 UTC, 0 replies.
- Dataset encoders for further types? - posted by Jakub Dubovsky <sp...@gmail.com> on 2016/12/15 16:18:47 UTC, 3 replies.
- Spark Batch checkpoint - posted by Selvam Raman <se...@gmail.com> on 2016/12/15 16:33:22 UTC, 4 replies.
- When will multiple aggregations be supported in Structured Streaming? - posted by ljwagerfield <la...@dmz.wagerfield.com> on 2016/12/15 18:43:09 UTC, 3 replies.
- [DataFrames] map function - 2.0 - posted by Ninad Shringarpure <ni...@cloudera.com> on 2016/12/15 21:17:43 UTC, 1 replies.
- Is there synchronous way to predict against model for real time data - posted by suyog choudhari <su...@gmail.com> on 2016/12/15 21:31:21 UTC, 2 replies.
- Re: Is restarting of SparkContext allowed? - posted by Marcelo Vanzin <va...@cloudera.com> on 2016/12/15 21:54:12 UTC, 0 replies.
- PowerIterationClustering Benchmark - posted by Lydia Ickler <ic...@googlemail.com> on 2016/12/15 22:04:27 UTC, 1 replies.
- Cached Tables SQL Performance Worse than Uncached - posted by Warren Kim <wk...@diablo-technologies.com> on 2016/12/15 22:14:45 UTC, 3 replies.
- Negative values of predictions in ALS.tranform - posted by Manish Tripathi <tr...@gmail.com> on 2016/12/15 22:20:54 UTC, 8 replies.
- Re: spark on yarn can't load kafka dependency jar - posted by neil90 <ne...@icloud.com> on 2016/12/15 22:49:55 UTC, 1 replies.
- Re: How to control saveAsTable() warehouse path? - posted by epettijohn <vi...@gmail.com> on 2016/12/15 22:57:47 UTC, 0 replies.
- Financial fraud detection using streaming RDBMS data into Spark & Hbase - posted by Mich Talebzadeh <mi...@gmail.com> on 2016/12/15 23:27:33 UTC, 0 replies.
- Why foreachPartition function make duplicate invocation to map function for every message ? (Spark 2.0.2) - posted by Michael Nguyen <ng...@gmail.com> on 2016/12/16 00:00:30 UTC, 1 replies.
- how to see cpu time of application on yarn mode - posted by 黄晓萌 <hu...@163.com> on 2016/12/16 02:07:09 UTC, 0 replies.
- How to reflect dynamic registration udf? - posted by 李斌松 <li...@gmail.com> on 2016/12/16 02:27:23 UTC, 1 replies.
- How to load edge with properties file useing GraphX - posted by "zjp_jdev@163.com" <zj...@163.com> on 2016/12/16 03:23:57 UTC, 1 replies.
- Spark Dataframe: Save to hdfs is taking long time - posted by KhajaAsmath Mohammed <md...@gmail.com> on 2016/12/16 03:54:27 UTC, 3 replies.
- Can't use RandomForestClassificationModel.predict(Vector v) in scala - posted by Patrick Chen <cz...@gmail.com> on 2016/12/16 06:00:22 UTC, 0 replies.
- Do we really need mesos or yarn? or is standalone sufficent? - posted by kant kodali <ka...@gmail.com> on 2016/12/16 08:14:08 UTC, 4 replies.
- What is the deployment model for Spark Streaming? A specific example. - posted by Russell Jurney <ru...@gmail.com> on 2016/12/16 08:53:39 UTC, 3 replies.
- Gradle dependency problem with spark - posted by kant kodali <ka...@gmail.com> on 2016/12/16 10:22:47 UTC, 4 replies.
- Spark dump in slave Node EMR - posted by Selvam Raman <se...@gmail.com> on 2016/12/16 12:30:32 UTC, 1 replies.
- unsuscribe - posted by Javier Rey <jr...@gmail.com> on 2016/12/16 14:42:08 UTC, 0 replies.
- java.lang.RuntimeException: Stream '/jars/' not found - posted by Hanumath Rao Maduri <ha...@gmail.com> on 2016/12/16 14:56:30 UTC, 0 replies.
- Issue: Skew on Dataframes while Joining the dataset - posted by KhajaAsmath Mohammed <md...@gmail.com> on 2016/12/16 15:51:33 UTC, 3 replies.
- Mesos Spark Fine Grained Execution - CPU count - posted by "Chawla,Sumit " <su...@gmail.com> on 2016/12/16 16:41:16 UTC, 18 replies.
- Spark GraphFrames generic question - posted by Ankur Srivastava <an...@gmail.com> on 2016/12/17 02:39:21 UTC, 0 replies.
- Regarding Connection Problem - posted by Chintan Bhatt <ch...@charusat.ac.in> on 2016/12/17 03:01:50 UTC, 1 replies.
- Running Multiple Versions of Spark on the same cluster (YARN) - posted by Jorge Machado <jo...@me.com> on 2016/12/17 07:36:50 UTC, 3 replies.
- need help to have a Java version of this scala script - posted by Richard Xin <ri...@yahoo.com.INVALID> on 2016/12/17 07:54:53 UTC, 2 replies.
- How to perform Join operation using JAVARDD - posted by Sree Eedupuganti <sr...@inndata.in> on 2016/12/17 10:49:58 UTC, 1 replies.
- Fwd: SparkLauncher does not return State/ID on a standalone cluster - posted by Rahul Raj <ra...@option3consulting.com> on 2016/12/17 11:22:35 UTC, 0 replies.
- theory question - posted by kant kodali <ka...@gmail.com> on 2016/12/17 12:33:33 UTC, 0 replies.
- The spark hive udf can read broadcast the variables? - posted by 李斌松 <li...@gmail.com> on 2016/12/17 15:24:11 UTC, 1 replies.
- Java to show struct field from a Dataframe - posted by Richard Xin <ri...@yahoo.com.INVALID> on 2016/12/17 15:28:09 UTC, 6 replies.
- foreachPartition's operation is taking long to finish - posted by Deepak Sharma <de...@gmail.com> on 2016/12/17 17:44:13 UTC, 4 replies.
- Kafka Spark structured streaming latency benchmark. - posted by Prashant Sharma <sc...@gmail.com> on 2016/12/18 03:19:38 UTC, 0 replies.
- Question about Spark and filesystems - posted by jo...@verona.se on 2016/12/18 19:50:17 UTC, 2 replies.
- [Spark SQL] Task failed while writing rows - posted by Joseph Naegele <jn...@grierforensics.com> on 2016/12/18 22:49:33 UTC, 4 replies.
- GraphFrame not init vertices when load edges - posted by "zjp_jdev@163.com" <zj...@163.com> on 2016/12/19 02:25:50 UTC, 4 replies.
- Reading xls and xlsx files - posted by Selvam Raman <se...@gmail.com> on 2016/12/19 10:12:21 UTC, 1 replies.
- How to set NameSpace while storing from Spark to HBase using saveAsNewAPIHadoopDataSet - posted by Rabin Banerjee <de...@gmail.com> on 2016/12/19 12:10:12 UTC, 2 replies.
- Spark SQL Syntax - posted by A Shaikh <sh...@gmail.com> on 2016/12/19 13:06:56 UTC, 1 replies.
- stratified sampling scales poorly - posted by Martin Le <ma...@gmail.com> on 2016/12/19 13:17:06 UTC, 0 replies.
- Re: Reference External Variables in Map Function (Inner class) - posted by mbayebabacar <mb...@outlook.com> on 2016/12/19 16:49:54 UTC, 0 replies.
- Adding Hive support to existing SparkSession (or starting PySpark with Hive support) - posted by apu <ap...@gmail.com> on 2016/12/19 17:58:16 UTC, 2 replies.
- Pivot in Spark with Case and when - posted by KhajaAsmath Mohammed <md...@gmail.com> on 2016/12/19 18:08:38 UTC, 0 replies.
- PySpark: [Errno 8] nodename nor servname provided, or not known - posted by "Jain, Nishit" <nj...@underarmour.com> on 2016/12/19 20:54:20 UTC, 1 replies.
- Loading a class from a dependency jar - posted by viraj <vi...@gmail.com> on 2016/12/19 22:51:39 UTC, 0 replies.
- How to deal with string column data for spark mlib? - posted by big data <bi...@outlook.com> on 2016/12/20 09:35:16 UTC, 5 replies.
- 答复: How to deal with string column data for spark mlib? - posted by "Triones,Deng(vip.com)" <tr...@vipshop.com> on 2016/12/20 12:35:50 UTC, 0 replies.
- 撤回: How to deal with string column data for spark mlib? - posted by "Triones,Deng(vip.com)" <tr...@vipshop.com> on 2016/12/20 12:36:20 UTC, 0 replies.
- question about the data frame save mode to make the data exactly one - posted by "Triones,Deng(vip.com)" <tr...@vipshop.com> on 2016/12/20 12:36:54 UTC, 0 replies.
- withColumn gives "Can only zip RDDs with same number of elements in each partition" but not with a LIMIT on the dataframe - posted by Jack Wenger <ja...@gmail.com> on 2016/12/20 16:04:25 UTC, 1 replies.
- Null pointer exception with RDD while computing a method, creating dataframe. - posted by satyajit vegesna <sa...@gmail.com> on 2016/12/21 00:47:26 UTC, 0 replies.
- scikit-learn and mllib difference in predictions python - posted by ioanna <gi...@hotmail.com> on 2016/12/21 01:21:41 UTC, 1 replies.
- access Broadcast Variables in Spark java - posted by Sateesh Karuturi <sa...@gmail.com> on 2016/12/21 06:25:20 UTC, 1 replies.
- Facing intermittent issue - posted by Manisha Sethi <Ma...@pb.com> on 2016/12/21 07:12:09 UTC, 0 replies.
- Launching multiple spark jobs within a main spark job. - posted by Naveen <ha...@gmail.com> on 2016/12/21 07:17:36 UTC, 7 replies.
- Approach: Incremental data load from HBASE - posted by Chetan Khatri <ch...@gmail.com> on 2016/12/21 10:28:58 UTC, 4 replies.
- SPARK -SQL Understanding BroadcastNestedLoopJoin and number of partitions - posted by David Hodeffi <Da...@niceactimize.com> on 2016/12/21 12:43:42 UTC, 1 replies.
- NoClassDefFoundError - posted by Reza zade <kn...@gmail.com> on 2016/12/21 13:02:55 UTC, 1 replies.
- Writing into parquet throws Array out of bounds exception - posted by Selvam Raman <se...@gmail.com> on 2016/12/21 13:37:46 UTC, 0 replies.
- Spark kryo serialization register Datatype[] - posted by geoHeil <ge...@gmail.com> on 2016/12/21 13:49:33 UTC, 2 replies.
- ML PIC - posted by Robert Hamilton <ro...@icloud.com> on 2016/12/21 15:05:08 UTC, 3 replies.
- [ On the use of Spark as 'storage system'] - posted by Enrico DUrso <en...@everis.com> on 2016/12/21 15:29:17 UTC, 1 replies.
- streaming performance - posted by "Mendelson, Assaf" <As...@rsa.com> on 2016/12/21 15:58:04 UTC, 2 replies.
- Parquet with group by queries - posted by Anil Langote <an...@gmail.com> on 2016/12/21 16:41:46 UTC, 1 replies.
- spark linear regression error training dataset is empty - posted by Xiaomeng Wan <sh...@gmail.com> on 2016/12/21 18:05:28 UTC, 1 replies.
- spark-shell fails to redefine values - posted by Yang <te...@gmail.com> on 2016/12/21 18:37:03 UTC, 1 replies.
- Has anyone managed to connect to Oracle via JDBC from Spark CDH 5.5.2 - posted by Mich Talebzadeh <mi...@gmail.com> on 2016/12/21 21:40:42 UTC, 6 replies.
- In PySpark ML, how can I interpret the SparseVector returned by a pyspark.ml.classification.RandomForestClassificationModel.featureImportances ? - posted by Russell Jurney <ru...@gmail.com> on 2016/12/22 00:11:26 UTC, 0 replies.
- Access HiveConf from SparkSession - posted by Vishak <vi...@gmail.com> on 2016/12/22 05:12:32 UTC, 1 replies.
- submit spark task on yarn asynchronously via java? - posted by Linyuxin <li...@huawei.com> on 2016/12/22 06:18:03 UTC, 0 replies.
- parsing embedded json in spark - posted by Tal Grynbaum <ta...@gmail.com> on 2016/12/22 07:42:20 UTC, 3 replies.
- Why does Spark 2.0 change number or partitions when reading a parquet file? - posted by Kristina Rogale Plazonic <kp...@gmail.com> on 2016/12/22 16:09:01 UTC, 1 replies.
- Ingesting data in elasticsearch from hdfs using spark , cluster setup and usage - posted by Rohit Verma <ro...@rokittech.com> on 2016/12/22 17:35:57 UTC, 3 replies.
- Spark subscribe - posted by pradeep s <sr...@gmail.com> on 2016/12/22 19:50:40 UTC, 0 replies.
- Merging Parquet Files - posted by Benjamin Kim <bb...@gmail.com> on 2016/12/22 22:01:28 UTC, 3 replies.
- 答复: submit spark task on yarn asynchronously via java? - posted by Linyuxin <li...@huawei.com> on 2016/12/23 01:40:50 UTC, 2 replies.
- Can't access the data in Kafka Spark Streaming globally - posted by Sree Eedupuganti <sr...@inndata.in> on 2016/12/23 05:46:23 UTC, 1 replies.
- Best Practice for Spark Job Jar Generation - posted by Chetan Khatri <ch...@gmail.com> on 2016/12/23 06:44:02 UTC, 4 replies.
- Is there any scheduled release date for Spark 2.1.0? - posted by Aseem Bansal <as...@gmail.com> on 2016/12/23 09:00:08 UTC, 8 replies.
- Dependency Injection and Microservice development with Spark - posted by Chetan Khatri <ch...@gmail.com> on 2016/12/23 10:56:47 UTC, 3 replies.
- ThreadPoolExecutor - slow spark job - posted by geoHeil <ge...@gmail.com> on 2016/12/23 13:19:52 UTC, 0 replies.
- Spark-SQL 1.6.2 w/Hive UDF @Description - posted by "Lavelle, Shawn" <Sh...@osii.com> on 2016/12/23 23:07:34 UTC, 0 replies.
- Getting Hive Metastore error without using Hive - posted by Léo Biscassi <le...@gmail.com> on 2016/12/25 15:53:15 UTC, 0 replies.
- Merri Christmas and Happy New Year - posted by ayan guha <gu...@gmail.com> on 2016/12/26 01:53:24 UTC, 0 replies.
- 答复: 答复: submit spark task on yarn asynchronously via java? - posted by Linyuxin <li...@huawei.com> on 2016/12/26 02:32:11 UTC, 0 replies.
- [Spark 2.0.2 HDFS]: no data locality - posted by Karamba <ph...@web.de> on 2016/12/26 17:39:25 UTC, 5 replies.
- Spark Storage Tab is empty - posted by David Hodeffi <Da...@niceactimize.com> on 2016/12/26 18:08:29 UTC, 1 replies.
- Spark Graphx with Database - posted by balaji9058 <ks...@gmail.com> on 2016/12/27 05:26:49 UTC, 1 replies.
- how can I get the application belong to the driver？ - posted by John Fang <xi...@alibaba-inc.com> on 2016/12/27 07:51:18 UTC, 0 replies.
- Location for the additional jar files in Spark - posted by Mich Talebzadeh <mi...@gmail.com> on 2016/12/27 09:33:33 UTC, 12 replies.
- Spark streaming with Yarn: executors not fully utilized - posted by Nishant Kumar <ni...@applift.com> on 2016/12/28 04:19:39 UTC, 2 replies.
- StackOverflow Error spark mllib 1.6.1 FpGrowth Algorithm for Association rule generation - posted by Maitray Thaker <mt...@adelement.com> on 2016/12/28 06:27:49 UTC, 0 replies.
- how to integrate Apache Kafka with spark ? - posted by sathyanarayanan mudhaliyar <sa...@gmail.com> on 2016/12/28 06:57:37 UTC, 1 replies.
- [MLlib] StackOverflow Error spark mllib 1.6.1 FpGrowth Algorithm for Association rule generation - posted by Maitray Thaker <mt...@adelement.com> on 2016/12/28 07:20:36 UTC, 0 replies.
- Apache Hive with Spark Configuration - posted by Chetan Khatri <ch...@gmail.com> on 2016/12/28 12:22:28 UTC, 1 replies.
- Getting values list per partition - posted by Rohit Verma <ro...@rokittech.com> on 2016/12/28 12:46:03 UTC, 0 replies.
- Error: at sqlContext.createDataFrame with RDD and Schema - posted by Chetan Khatri <ch...@gmail.com> on 2016/12/28 15:30:55 UTC, 1 replies.
- [PySpark - 1.6] - Avoid object serialization - posted by Sidney Feiner <si...@startapp.com> on 2016/12/28 15:57:26 UTC, 3 replies.
- Spark/Mesos with GPU support - posted by Ji Yan <ji...@drive.ai> on 2016/12/28 19:22:19 UTC, 4 replies.
- org.apache.spark.SparkException: PairwiseRDD: unexpected value: List([B@20b7e9d2) - posted by prayag <pr...@gmail.com> on 2016/12/28 22:58:22 UTC, 0 replies.
- Invert large matrix - posted by Yanwei Wayne Zhang <ac...@hotmail.com> on 2016/12/29 02:22:26 UTC, 0 replies.
- [TorrentBroadcast] Pyspark Application terminated saying "Failed to get broadcast_1_ piece0 of broadcast_1 in Spark 2.0.0" - posted by Palash Gupta <sp...@yahoo.com.INVALID> on 2016/12/29 06:25:58 UTC, 11 replies.
- Re: Jdbc connection from sc.addJar failing - posted by drtrotter74 <d....@charter.net> on 2016/12/29 08:02:54 UTC, 0 replies.
- spark program for dependent cascading operations - posted by srimugunthan dhandapani <sr...@gmail.com> on 2016/12/29 10:45:27 UTC, 0 replies.
- [Spark Structured Streaming]: Is it possible to ingest data from a jdbc data source incrementally? - posted by "Yuanzhe Yang (杨远哲)" <yy...@gmail.com> on 2016/12/29 12:05:50 UTC, 3 replies.
- send this email to unsubscribe - posted by John <jo...@ifrec.osaka-u.ac.jp> on 2016/12/29 12:58:18 UTC, 0 replies.
- [ANNOUNCE] Announcing Apache Spark 2.1.0 - posted by Yin Huai <yh...@databricks.com> on 2016/12/29 16:02:59 UTC, 2 replies.
- Issue with SparkR setup on RStudio - posted by "Md. Rezaul Karim" <re...@insight-centre.org> on 2016/12/29 18:24:57 UTC, 1 replies.
- Reading specific column family and columns in Hbase table through spark - posted by Mich Talebzadeh <mi...@gmail.com> on 2016/12/29 20:12:18 UTC, 0 replies.
- Cosine Similarity of Word2Vec algo more than 1? - posted by Manish Tripathi <tr...@gmail.com> on 2016/12/29 21:48:22 UTC, 5 replies.
- Re: Invert large matrix - posted by Sean Owen <so...@cloudera.com> on 2016/12/29 21:52:12 UTC, 1 replies.
- Best way to process lookup ETL with Dataframes - posted by "Sesterhenn, Mike" <ms...@cars.com> on 2016/12/29 22:55:26 UTC, 5 replies.
- DataFrame to read json and include raw Json in DataFrame - posted by Richard Xin <ri...@yahoo.com.INVALID> on 2016/12/30 01:16:24 UTC, 3 replies.
- RDD Location - posted by Fei Hu <hu...@gmail.com> on 2016/12/30 04:06:39 UTC, 4 replies.
- [Spark streaming 1.6.0] Spark streaming with Yarn: executors not fully utilized - posted by Nishant Kumar <ni...@applift.com> on 2016/12/30 04:53:25 UTC, 0 replies.
- Re: Spark Partitioning Strategy with Parquet - posted by titli batali <ti...@gmail.com> on 2016/12/30 08:23:27 UTC, 0 replies.
- Difference in R and Spark Output - posted by Saroj C <sa...@tcs.com> on 2016/12/30 13:12:04 UTC, 1 replies.
- Broadcast Join and Inner Join giving different result on same DataFrame - posted by titli batali <ti...@gmail.com> on 2016/12/30 15:45:20 UTC, 0 replies.
- Broadcast destroy - posted by br...@gmail.com on 2016/12/30 18:20:02 UTC, 0 replies.
- What's the best practice to load data from RDMS to Spark - posted by Raymond Xie <xi...@gmail.com> on 2016/12/30 20:09:59 UTC, 1 replies.
- launch spark on mesos within a docker container - posted by Ji Yan <ji...@drive.ai> on 2016/12/30 21:23:58 UTC, 3 replies.
- context.runJob() was suspended in getPreferredLocations() function - posted by Fei Hu <hu...@gmail.com> on 2016/12/31 01:55:10 UTC, 0 replies.
- How to load a big csv to dataframe in Spark 1.6 - posted by Raymond Xie <xi...@gmail.com> on 2016/12/31 02:46:11 UTC, 7 replies.
- [ML] Converting ml.DenseVector to mllib.Vector - posted by Jason Wolosonovich <jm...@asu.edu> on 2016/12/31 04:24:06 UTC, 2 replies.
- New runtime exception after switch to Spark 2.1.0 - posted by mhornbech <mo...@datasolvr.com> on 2016/12/31 12:34:33 UTC, 0 replies.
- Custom delimiter file load - posted by A Shaikh <sh...@gmail.com> on 2016/12/31 17:58:10 UTC, 1 replies.
- From Hive to Spark, what is the default database/table - posted by Raymond Xie <xi...@gmail.com> on 2016/12/31 20:55:11 UTC, 1 replies.