user@spark.apache.org, 2019-05

You are viewing a plain text version of this content. The canonical link for it is here.

- RE: [EXT] handling skewness issues - posted by em...@yeikel.com on 2019/05/01 02:25:14 UTC, 0 replies.
- RE: Turning off Jetty Http Options Method - posted by em...@yeikel.com on 2019/05/01 02:35:41 UTC, 1 replies.
- spark on kubernetes driver pod phase changed from running to pending and starts another container in pod - posted by zyfo2 <zh...@gmail.com> on 2019/05/01 02:59:04 UTC, 0 replies.
- Error while using spark-avro module in pyspark 2.4 - posted by kanchan tewary <ka...@gmail.com> on 2019/05/01 07:55:28 UTC, 0 replies.
- Re: Spark Structured Streaming | Highly reliable de-duplication strategy - posted by Akshay Bhardwaj <ak...@gmail.com> on 2019/05/01 09:15:25 UTC, 2 replies.
- Re: Update / Delete records in Parquet - posted by Vitaliy Pisarev <vi...@biocatch.com> on 2019/05/01 09:20:33 UTC, 1 replies.
- Re: Best notebook for developing for apache spark using scala on Amazon EMR Cluster - posted by Jeff Zhang <zj...@gmail.com> on 2019/05/01 13:27:46 UTC, 0 replies.
- Spark SQL LIMIT Gets Stuck - posted by Shahab Yunus <sh...@gmail.com> on 2019/05/01 13:50:04 UTC, 0 replies.
- common logging in spark - posted by rajat kumar <ku...@gmail.com> on 2019/05/01 18:07:32 UTC, 0 replies.
- Re: spark df.write.partitionBy run very slow - posted by Shyam P <sh...@gmail.com> on 2019/05/02 07:24:00 UTC, 0 replies.
- pySpark - pandas UDF and binaryType - posted by Nicolas Paris <ni...@riseup.net> on 2019/05/02 11:25:41 UTC, 4 replies.
- What is Spark context cleaner in structured streaming - posted by Akshay Bhardwaj <ak...@gmail.com> on 2019/05/02 11:51:04 UTC, 1 replies.
- Spark SQL Teradata load is very slow - posted by KhajaAsmath Mohammed <md...@gmail.com> on 2019/05/02 17:59:38 UTC, 1 replies.
- Re: Spark 2.4.1 on Kubernetes - DNS resolution of driver fails - posted by Li Gao <li...@gmail.com> on 2019/05/03 01:06:40 UTC, 1 replies.
- Re: Getting EOFFileException while reading from sequence file in spark - posted by Prateek Rajput <pr...@flipkart.com.INVALID> on 2019/05/03 09:44:27 UTC, 0 replies.
- Spark SQL JDBC teradata syntax error - posted by KhajaAsmath Mohammed <md...@gmail.com> on 2019/05/03 16:28:08 UTC, 1 replies.
- Howto force spark to honor parquet partitioning - posted by Tomas Bartalos <to...@gmail.com> on 2019/05/03 16:38:22 UTC, 1 replies.
- [MLlib][Beginner][Debug]: Logistic Regression model always predicts the same value - posted by Josue Lopes <jo...@randoli.ca> on 2019/05/03 19:11:21 UTC, 0 replies.
- error when running decisiontree in java - posted by Serena S Yuan <su...@gmail.com> on 2019/05/04 06:01:00 UTC, 0 replies.
- Dynamic metric names - posted by Sergey Zhemzhitsky <sz...@gmail.com> on 2019/05/04 08:44:15 UTC, 4 replies.
- Deep Learning with Spark, what is your experience? - posted by Riccardo Ferrari <fe...@gmail.com> on 2019/05/04 09:51:41 UTC, 9 replies.
- Re: This MapR-DB Spark Connector with Secondary Indexes - posted by Mich Talebzadeh <mi...@gmail.com> on 2019/05/04 23:03:10 UTC, 0 replies.
- Re: Anaconda installation with Pyspark/Pyarrow (2.3.0+) on cloudera managed server - posted by Rishi Shah <ri...@gmail.com> on 2019/05/05 02:41:56 UTC, 11 replies.
- write files of a specific size - posted by rajat kumar <ku...@gmail.com> on 2019/05/05 08:51:06 UTC, 2 replies.
- Request for a working example of using Pregel API in GraphX using Spark Scala - posted by Basavaraj <ra...@gmail.com> on 2019/05/05 08:52:51 UTC, 0 replies.
- batch processing in spark - posted by swastik mittal <sm...@ncsu.edu> on 2019/05/05 18:59:26 UTC, 1 replies.
- K8S Spark submit - posted by "Ben Chukwumobi (CONT)" <if...@capitalone.com> on 2019/05/06 02:12:59 UTC, 0 replies.
- K8S spark submit for spark 2.4 - posted by "Ben Chukwumobi (CONT)" <if...@capitalone.com> on 2019/05/06 02:30:37 UTC, 0 replies.
- Performance Decrease in spark - posted by yuvraj singh <19...@gmail.com> on 2019/05/06 07:27:40 UTC, 1 replies.
- Spark structured streaming watermarks on nested attributes - posted by Joe Ammann <jo...@pyx.ch> on 2019/05/06 13:44:48 UTC, 4 replies.
- Image Grep - posted by swastik mittal <sm...@ncsu.edu> on 2019/05/06 21:45:54 UTC, 0 replies.
- spark-cassandra-connector_2.1 caused java.lang.NoClassDefFoundError under Spark 2.4.2? - posted by Richard Xin <ri...@yahoo.com.INVALID> on 2019/05/07 00:53:29 UTC, 0 replies.
- Re: spark-cassandra-connector_2.1 caused java.lang.NoClassDefFoundError under Spark 2.4.2? - posted by Russell Spitzer <ru...@gmail.com> on 2019/05/07 01:34:22 UTC, 4 replies.
- Re: Spark SQL met "Block broadcast_xxx not found" - posted by Xilang Yan <xi...@gmail.com> on 2019/05/07 07:34:50 UTC, 1 replies.
- ThriftServer gc over exceed and memory problem - posted by "shicheng31604@gmail.com" <sh...@gmail.com> on 2019/05/07 13:31:28 UTC, 0 replies.
- Static partitioning in partitionBy() - posted by Shubham Chaurasia <sh...@gmail.com> on 2019/05/07 13:36:02 UTC, 4 replies.
- Structured Streaming Kafka - Weird behavior with performance and logs - posted by Austin Weaver <au...@flyrlabs.com> on 2019/05/07 14:32:08 UTC, 4 replies.
- Running Spark 2.4 on K8S - posted by Bigg Ben <ch...@gmail.com> on 2019/05/07 14:36:53 UTC, 1 replies.
- Create table from Avro-generated parquet files? - posted by "Coolbeth, Matthew" <Ma...@espn.com> on 2019/05/07 20:18:39 UTC, 0 replies.
- How does org.apache.spark.sql.catalyst.util.MapData support hash lookup? - posted by Shawn Yang <sh...@gmail.com> on 2019/05/08 09:25:50 UTC, 0 replies.
- HiveTableRelation Assertion Error | Joining Stream with Hive table - posted by Manjunath N <ma...@gmail.com> on 2019/05/08 11:10:02 UTC, 0 replies.
- IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff] while using spark-sql-2.4.1v to read data from oracle - posted by Shyam P <sh...@gmail.com> on 2019/05/08 14:14:14 UTC, 0 replies.
- pyspark on pycharm error - posted by karan alang <ka...@gmail.com> on 2019/05/09 00:09:39 UTC, 0 replies.
- Train ML models on each partition - posted by Qian He <hq...@gmail.com> on 2019/05/09 05:27:50 UTC, 1 replies.
- Generic Dataset[T] Query - posted by SNEHASISH DUTTA <in...@gmail.com> on 2019/05/09 06:03:38 UTC, 1 replies.
- [ANNOUNCE] Announcing Apache Spark 2.4.3 - posted by Xiao Li <ga...@gmail.com> on 2019/05/09 06:56:49 UTC, 0 replies.
- The following Java MR code works for small dataset but throws(arrayindexoutofBound) error for large dataset - posted by Balakumar iyer S <ba...@gmail.com> on 2019/05/09 09:03:20 UTC, 1 replies.
- How to configure alluxio cluster with spark in yarn - posted by u9g <lw...@163.com> on 2019/05/09 11:18:51 UTC, 1 replies.
- How to fix ClosedChannelException - posted by u9g <lw...@163.com> on 2019/05/09 11:22:39 UTC, 1 replies.
- Spark foreach retry - posted by hemant singh <he...@gmail.com> on 2019/05/09 11:48:16 UTC, 0 replies.
- Question about SaveMode.Ignore behaviour - posted by Juho Autio <ju...@rovio.com> on 2019/05/09 13:28:31 UTC, 1 replies.
- Spark not doing a broadcast join inspite of the table being well below spark.sql.autoBroadcastJoinThreshold - posted by V0lleyBallJunki3 <ve...@gmail.com> on 2019/05/09 14:19:30 UTC, 1 replies.
- Spark Elasticsearch Connector | Index and Update - posted by Akshay Bhardwaj <ak...@gmail.com> on 2019/05/10 06:52:27 UTC, 0 replies.
- Spark on yarn - application hangs - posted by Mkal <di...@hotmail.com> on 2019/05/10 16:50:40 UTC, 3 replies.
- ml Pipeline read write - posted by Koert Kuipers <ko...@tresata.com> on 2019/05/10 20:31:06 UTC, 1 replies.
- spark error when initializing spark session in java - posted by Serena S Yuan <su...@gmail.com> on 2019/05/10 21:01:26 UTC, 0 replies.
- spark.sql.autoBroadcastJoinThreshold not taking effect - posted by V0lleyBallJunki3 <ve...@gmail.com> on 2019/05/11 01:54:27 UTC, 1 replies.
- log level in spark - posted by Nicolas Paris <ni...@riseup.net> on 2019/05/11 12:54:49 UTC, 1 replies.
- Streaming job, catch exceptions - posted by Behroz Sikander <be...@gmail.com> on 2019/05/11 19:43:50 UTC, 12 replies.
- Kafka Spark Streaming integration : Relationship between DStreams and Tasks - posted by Sheel Pancholi <sh...@gmail.com> on 2019/05/12 18:58:45 UTC, 1 replies.
- spark jar version problem - posted by "wangyongqiang0617@163.com" <wa...@163.com> on 2019/05/13 07:14:43 UTC, 0 replies.
- Spark Fair Scheduler does not work correctly - posted by Yuta Morisawa <yu...@kddi-research.jp> on 2019/05/13 08:44:51 UTC, 0 replies.
- [Pyspark 2.3] Logical operators (and/or) in pyspark - posted by Rishi Shah <ri...@gmail.com> on 2019/05/13 15:37:37 UTC, 1 replies.
- [pyspark 2.3] drop_duplicates, keep first record based on sorted records - posted by Rishi Shah <ri...@gmail.com> on 2019/05/13 16:41:32 UTC, 0 replies.
- Getting List of Executor Id's - posted by "Afshartous, Nick" <na...@wbgames.com> on 2019/05/13 18:26:27 UTC, 1 replies.
- Error using .collect() - posted by Kumar sp <kr...@gmail.com> on 2019/05/13 19:49:20 UTC, 1 replies.
- Handling of watermark in structured streaming - posted by Joe Ammann <jo...@pyx.ch> on 2019/05/14 13:48:39 UTC, 6 replies.
- Why do we need Java-Friendly APIs in Spark ? - posted by Gary Gao <ga...@gmail.com> on 2019/05/14 14:22:11 UTC, 4 replies.
- What is the difference for the following UDFs? - posted by Qian He <hq...@gmail.com> on 2019/05/14 16:08:16 UTC, 2 replies.
- Re: BigDL and Analytics Zoo talks at upcoming Spark+AI Summit and Strata London - posted by Jason Dai <ja...@gmail.com> on 2019/05/15 01:05:58 UTC, 0 replies.
- Re: Koalas show data in IDE or pyspark - posted by Reynold Xin <rx...@databricks.com> on 2019/05/15 02:01:02 UTC, 0 replies.
- In spark 2.4 Upgrade Apache Arrow to version 0.12.0 - posted by 李斌松 <li...@gmail.com> on 2019/05/15 03:10:07 UTC, 0 replies.
- Spark job gets hung on cloudera cluster - posted by Rishi Shah <ri...@gmail.com> on 2019/05/15 03:51:21 UTC, 5 replies.
- Databricks - number of executors, shuffle.partitions etc - posted by Rishi Shah <ri...@gmail.com> on 2019/05/15 03:52:43 UTC, 3 replies.
- Spark sql insert hive table which method has the highest performance - posted by 车 <si...@outlook.com> on 2019/05/15 06:25:40 UTC, 1 replies.
- Running spark with javaagent configuration - posted by Anton Puzanov <an...@gmail.com> on 2019/05/15 13:27:58 UTC, 2 replies.
- Are Spark Dataframes mutable in Structured Streaming? - posted by Sheel Pancholi <sh...@gmail.com> on 2019/05/15 18:25:06 UTC, 12 replies.
- how to get spark-sql lineage - posted by lk_spark <lk...@163.com> on 2019/05/16 02:44:31 UTC, 2 replies.
- [Structured Streaming]: Are Spark Dataframes mutable in Structured Streaming? - posted by Sheel Pancholi <sh...@gmail.com> on 2019/05/16 09:55:47 UTC, 0 replies.
- GC overhead while read a table partition from HIVE - posted by Shivam Sharma <28...@gmail.com> on 2019/05/16 13:08:01 UTC, 0 replies.
- GraphX parameters tuning - posted by muaz-32 <mu...@euranova.eu> on 2019/05/16 15:20:23 UTC, 0 replies.
- Is there a difference between --proxy-user or HADOOP_USER_NAME in a non-Kerberized YARN cluster? - posted by Jeff Evans <je...@gmail.com> on 2019/05/16 17:49:02 UTC, 0 replies.
- design question related to kafka. - posted by Shyam P <sh...@gmail.com> on 2019/05/17 07:11:11 UTC, 0 replies.
- Spark streaming - posted by Antoine DUBOIS <an...@cc.in2p3.fr> on 2019/05/17 12:28:32 UTC, 0 replies.
- Out Of Memory while reading a table partition from HIVE - posted by Shivam Sharma <28...@gmail.com> on 2019/05/17 15:23:25 UTC, 0 replies.
- Access to live data of cached dataFrame - posted by Tomas Bartalos <to...@gmail.com> on 2019/05/17 18:37:34 UTC, 3 replies.
- [PSA] Sharing our Experiences With Kubernetes - posted by Matt Cheah <mc...@palantir.com> on 2019/05/17 19:22:04 UTC, 1 replies.
- Fwd: Spark Architecture, Drivers, & Executors - posted by Pat Ferrel <pa...@occamsmachete.com> on 2019/05/17 19:26:58 UTC, 0 replies.
- Spark-YARN | Scheduling of containers - posted by Akshay Bhardwaj <ak...@gmail.com> on 2019/05/19 18:55:16 UTC, 4 replies.
- [spark on yarn] spark on yarn without DFS - posted by Huizhe Wang <wa...@husky.neu.edu> on 2019/05/20 01:50:04 UTC, 7 replies.
- spark 2.4.3 build fails using java 8 and scala 2.11 with NumberFormatException: Not a version: 9 - posted by Bulldog20630405 <bu...@gmail.com> on 2019/05/20 02:22:37 UTC, 1 replies.
- Watermark handling on initial query start (Structured Streaming) - posted by Joe Ammann <jo...@pyx.ch> on 2019/05/20 09:49:57 UTC, 0 replies.
- Fetching LinkedIn data into PySpark using OAuth2.0 - posted by Aakash Basu <aa...@gmail.com> on 2019/05/20 12:45:04 UTC, 0 replies.
- run new spark version on old spark cluster ? - posted by Nicolas Paris <ni...@riseup.net> on 2019/05/20 17:50:29 UTC, 9 replies.
- High level explanation of dropDuplicates - posted by Yeikel <em...@yeikel.com> on 2019/05/20 18:31:55 UTC, 1 replies.
- [pyspark 2.3] count followed by write on dataframe - posted by Rishi Shah <ri...@gmail.com> on 2019/05/21 01:43:05 UTC, 1 replies.
- spark checkpoint between 2 jobs and HDFS ramfs with storage policy - posted by Julien Laurenceau <ju...@pepitedata.com> on 2019/05/21 08:03:11 UTC, 0 replies.
- How does dynamic allocation decide spark executor cores? - posted by Pooja Agrawal <po...@gmail.com> on 2019/05/21 10:03:16 UTC, 0 replies.
- NoClassDefFoundError - posted by Sachit Murarka <co...@gmail.com> on 2019/05/21 11:11:04 UTC, 0 replies.
- double quota is automaticly added when sinking as csv - posted by 杨浩 <ya...@gmail.com> on 2019/05/21 12:03:43 UTC, 1 replies.
- Structred Streaming Error - posted by KhajaAsmath Mohammed <md...@gmail.com> on 2019/05/21 19:15:46 UTC, 2 replies.
- Offsets out of order - Spark Datasource V2 - posted by "Cressy, Taylor" <Ta...@intuit.com> on 2019/05/21 20:48:03 UTC, 0 replies.
- [pyspark 2.3+] how to dynamically determine DataFrame partitions while writing - posted by Rishi Shah <ri...@gmail.com> on 2019/05/22 03:29:16 UTC, 1 replies.
- How does number of partitions in DataFrame get decided while reading from HIVE - posted by Shivam Sharma <28...@gmail.com> on 2019/05/22 10:05:32 UTC, 0 replies.
- [pyspark 2.3+] repartition followed by window function - posted by Rishi Shah <ri...@gmail.com> on 2019/05/22 10:32:01 UTC, 1 replies.
- [Spark K8] Kube2Iam Annotation Support - posted by Chandu Kavar <cc...@gmail.com> on 2019/05/22 13:55:37 UTC, 0 replies.
- Java heap error - posted by Kumar sp <kr...@gmail.com> on 2019/05/22 18:09:40 UTC, 0 replies.
- Specifying yarn queue w/ Livy Controller Service - posted by Varun Rao <va...@gmail.com> on 2019/05/22 19:47:36 UTC, 0 replies.
- Similar Narrow Transformations should be chanined? - posted by nicks29 <sh...@gmail.com> on 2019/05/22 22:42:31 UTC, 0 replies.
- PySpark Streaming “PicklingError: Could not serialize object” when use transform operator and checkpoint enabled - posted by Xilang Yan <xi...@gmail.com> on 2019/05/23 12:51:52 UTC, 0 replies.
- Executors idle, driver heap exploding and maxing only 1 cpu core - posted by Ashic Mahtab <as...@live.com> on 2019/05/23 14:36:14 UTC, 2 replies.
- unsubscribe - posted by "Mun, Woyou - US" <mi...@caci.com> on 2019/05/23 15:31:08 UTC, 1 replies.
- Spark hive table dependent on parquet version too low - posted by 李斌松 <li...@gmail.com> on 2019/05/24 07:35:32 UTC, 0 replies.
- unsubcribe - posted by Nikhil R Patil <Ni...@ril.com> on 2019/05/24 09:49:11 UTC, 0 replies.
- Writing to multiple Kafka partitions from Spark - posted by Femi Anthony <fe...@gmail.com> on 2019/05/24 15:34:46 UTC, 1 replies.
- Spark standalone and Pandas UDF from custom archive - posted by Riccardo Ferrari <fe...@gmail.com> on 2019/05/24 20:24:18 UTC, 3 replies.
- Does Spark SQL has match_recognize? - posted by kant kodali <ka...@gmail.com> on 2019/05/25 19:55:47 UTC, 2 replies.
- conflict with multiple jobs writing to different partitions but same baseDir - posted by Koert Kuipers <ko...@tresata.com> on 2019/05/25 22:04:02 UTC, 0 replies.
- Scaling Kafka Streaming to Thousands of Partitions - posted by Charles Chao <xp...@gmail.com> on 2019/05/25 23:32:42 UTC, 0 replies.
- Spark 2.4.3 on Kubernetes Client mode fails - posted by David Aspegren <da...@gmail.com> on 2019/05/26 11:41:50 UTC, 0 replies.
- How to improve binaryFiles performance? - posted by big data <bi...@outlook.com> on 2019/05/27 08:58:04 UTC, 0 replies.
- 1 task per executor - posted by Sachit Murarka <co...@gmail.com> on 2019/05/28 08:05:28 UTC, 1 replies.
- Putting record in HBase with Spark - error get regions. - posted by Guillermo Ortiz Fernández <gu...@gmail.com> on 2019/05/28 10:12:28 UTC, 1 replies.
- Phrase Search using Apache Spark in huge amount of text in files - posted by Sandeep Giri <sa...@knowbigdata.com> on 2019/05/28 14:27:56 UTC, 0 replies.
- [Spark Streaming]: Spark Checkpointing: Content, Recovery and Idempotency - posted by Sheel Pancholi <sh...@gmail.com> on 2019/05/29 15:40:41 UTC, 0 replies.
- adding a column to a groupBy (dataframe) - posted by Marcelo Valle <ma...@ktech.com> on 2019/05/29 16:11:24 UTC, 0 replies.
- Upsert for hive tables - posted by Tomasz Krol <pa...@gmail.com> on 2019/05/29 16:41:10 UTC, 6 replies.
- Spark Streaming: Checkpointing Recovery and Idempotency - posted by Sheel Pancholi <sh...@gmail.com> on 2019/05/29 17:47:04 UTC, 0 replies.
- Spark Streaming: Checkpoint, Recovery and Idempotency - posted by sheelstera <sh...@gmail.com> on 2019/05/29 17:55:03 UTC, 0 replies.
- Re: Should python-2 be supported in Spark 3.0? - posted by Xiangrui Meng <me...@gmail.com> on 2019/05/30 04:46:54 UTC, 7 replies.
- dynamic allocation in spark-shell - posted by Qian He <hq...@gmail.com> on 2019/05/31 00:00:12 UTC, 1 replies.
- [pyspark 2.3+] Bucketing with sort - incremental data load? - posted by Rishi Shah <ri...@gmail.com> on 2019/05/31 02:43:05 UTC, 2 replies.
- java.util.NoSuchElementException: Columns not found - posted by Shyam P <sh...@gmail.com> on 2019/05/31 11:04:03 UTC, 0 replies.