Spark 1.5.2, DataFrame broadcast join, OOM

[Yong Zhang <ja...@hotmail.com>]

Hi, 
I am testing the Spark 1.5.2 using a 4 nodes cluster, with 64G memory each, and one is master and 3 are workers. I am using Standalone mode.
Here is my spark-env.sh for settings:
export SPARK_LOCAL_DIRS=/data1/spark/local,/data2/spark/local,/data3/spark/localexport SPARK_MASTER_WEBUI_PORT=8081export SPARK_MASTER_IP=hostnameexport SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=3"export SPARK_WORKER_MEMORY=24gexport SPARK_WORKER_CORES=6export SPARK_WORKER_DIR=/tmp/spark/workexport SPARK_DRIVER_MEMORY=4gexport SPARK_EXECUTOR_MEMORY=2gexport SPARK_DAEMON_MEMORY=1g
I start my spark-shell in the following command: /opt/spark/bin/spark-shell --conf spark-executor_memory=8g --conf total_executor_cores=12
So each executor will have 8G as total heap, and allow up to 12/3=4 concurrence running.
Now I have a join between 2 dataframes:
val loadRaw = sqlContext.read.parquet("parquet_file_path")val trialRaw = loadRaw.filter(instr(loadRaw("event_list"), "202") > 0).select("soid_e1","visid_high","visid_low","date_time")
// trialRaw.count is 1106, with 4 columns, very small data
val historyRaw = sqlContext.read.parquet("multi path parquet files")
// historyRaw.count is 13.5M, but with 500 columns
If I run the join this way:
val join1 = trialRaw.join(historyRaw, trialRaw("visid_high") <=> historyRaw("visid_high") &&  trialRaw("visid_low") <=> historyRaw("visid_low") && trialRaw("date_time") > historyRaw("date_time"))join1.count
I can get the result after 20 minutes, in the above setting on this cluster. But I know the best way is to do the broadcast join, as the trailRaw is so small, so I did this:
val join2 = historyRaw.join(broadcast(trialRaw), trialRaw("visid_high") <=> historyRaw("visid_high") &&  trialRaw("visid_low") <=> historyRaw("visid_low") && trialRaw("date_time") > historyRaw("date_time"))join2.count
To my surprise, in this case the executors got the java.lang.OutOfMemoryError: GC overhead limit exceeded, I know the historyRaw is around 500 columns, which is big. My end result does want to include all the 500 columns in the join, as long as the join conditions match. But the count of row will be limited. Also the trialRaw is very small, only 1106 rows with 4 columns. This is the only data should be shipped to all the executors, and they just read the Parquet data and match by the join conditions.Why in this case it runs out of memory?
1) I know the OOM is not in Driver, as it has 4G heap, and the OOM error comes from the executor stderr2) I know I can add more heap, but I really want to know is that under the same settings, why join1 works, but join2 failed with such small data broadcasting?3) Is this related to how the parquet files being read by Spark?4) It is my first time trying this broadcast join in Spark dataframe, as it is only available since 1.5. So maybe there is some pitfall I don't understand how it works.
This is the stdout, it shows the full GC, then OOM:
2016-02-22T11:54:44.014-0500: [Full GC [PSYoungGen: 1994715K->1943403K(2395136K)] [ParOldGen: 5592413K->5592495K(5592576K)] 7587129K->7535898K(7987712K) [PSPermGen: 63554K->63548K(64000K)], 3.7543230 secs] [Times: user=50.72 sys=0.06, real=3.75 secs] 
2016-02-22T11:54:47.803-0500: [Full GC [PSYoungGen: 1994752K->1960598K(2395136K)] [ParOldGen: 5592495K->5592485K(5592576K)] 7587247K->7553083K(7987712K) [PSPermGen: 63690K->63587K(64000K)], 30.9218790 secs] [Times: user=543.07 sys=2.50, real=30.91 secs] 
Heap
 PSYoungGen      total 2395136K, used 1988191K [0x0000000755500000, 0x0000000800000000, 0x0000000800000000)
  eden space 1994752K, 99% used [0x0000000755500000,0x00000007cea97c30,0x00000007cf100000)
  from space 400384K, 0% used [0x00000007cf100000,0x00000007cf100000,0x00000007e7800000)
  to   space 390656K, 0% used [0x00000007e8280000,0x00000007e8280000,0x0000000800000000)
 ParOldGen       total 5592576K, used 5592485K [0x00000005fff80000, 0x0000000755500000, 0x0000000755500000)
  object space 5592576K, 99% used [0x00000005fff80000,0x00000007554e94e8,0x0000000755500000)
 PSPermGen       total 64000K, used 63710K [0x00000005eff80000, 0x00000005f3e00000, 0x00000005fff80000)
  object space 64000K, 99% used [0x00000005eff80000,0x00000005f3db7ac8,0x00000005f3e00000)
This is the executor log:16/02/22 11:53:41 INFO hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 0 records.
16/02/22 11:54:27 INFO parquet.ParquetRelation$$anonfun$buildScan$1$$anon$1: Input split: ParquetInputSplit{part: hdfs://host:9000/data/event_parquet/2015/08/21/event_parquet-2015-08-21-20150831195822.parquet start: 134217728 end: 268435456 length: 134217728 hosts: []}
16/02/22 11:54:42 WARN hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
16/02/22 11:54:42 ERROR executor.Executor: Managed memory leak detected; size = 33554432 bytes, TID = 206
16/02/22 11:54:42 ERROR executor.Executor: Exception in task 1.0 in stage 4.0 (TID 206)
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:133)
	at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
	at java.io.Writer.write(Writer.java:157)
Below is the join plan:
scala> join2.explain16/02/23 17:11:03 INFO storage.MemoryStore: ensureFreeSpace(156640) called with curMem=252597, maxMem=222302306316/02/23 17:11:03 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 153.0 KB, free 2.1 GB)16/02/23 17:11:03 INFO storage.MemoryStore: ensureFreeSpace(16125) called with curMem=409237, maxMem=222302306316/02/23 17:11:03 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 15.7 KB, free 2.1 GB)16/02/23 17:11:03 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on 10.20.180.71:46082 (size: 15.7 KB, free: 2.1 GB)16/02/23 17:11:03 INFO spark.SparkContext: Created broadcast 5 from explain at <console>:2816/02/23 17:11:03 INFO storage.MemoryStore: ensureFreeSpace(156640) called with curMem=425362, maxMem=222302306316/02/23 17:11:03 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 153.0 KB, free 2.1 GB)16/02/23 17:11:03 INFO storage.MemoryStore: ensureFreeSpace(16125) called with curMem=582002, maxMem=222302306316/02/23 17:11:03 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 15.7 KB, free 2.1 GB)16/02/23 17:11:03 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on 10.20.180.71:46082 (size: 15.7 KB, free: 2.1 GB)16/02/23 17:11:03 INFO spark.SparkContext: Created broadcast 6 from explain at <console>:28== Physical Plan ==Filter (((visid_high#460L <=> visid_high#948L) && (visid_low#461L <=> visid_low#949L)) && (date_time#25L > date_time#513L)) CartesianProduct  Scan ParquetRelation[hdfs://xxx/data/event_parquet/2015/08/13,hdfs://xxx/data/event_parquet/2015/08/14,hdfs://xxx/data/event_parquet/2015/08/21][accept_language#488,browser#489L,browser_height#490,browser_width#491,campaign#492,c_color#493,channel#494,click_action#495,click_action_type#496,click_context#497,click_context_type#498,click_sourceId#499L,click_tag#500,code_ver#501,color#502,connection_type#503,cookies#504,country#505,ct_connect_type#506,currency#507,curr_factor#508,curr_rate#509,cust_hit_time_gmt#510L,cust_visid#511,daily_visitor#512,date_time#513L,domain#514,duplicated_from#515,duplicate_events#516,duplicate_purchase#517,soid_e1#518,lpuid_e2#519,customer_status_e3#520,partner_uid_e4#521,ab_split_group_e5#522,application_e6#523,page_name_e7#524,originating_page_e8#525,email_signup_e9#526,survey_signup_e10#527,event_signup_e11#528,email_paying_e12#529,survey_paying_e13#530,event_paying_e14#531,image_hosting_paying_e15#532,archive_paying_e16#533,internal_search_query_e17#534,internal_search_page_e18#535,download_file_name_e19#536,download_file_page_e20#537,campaign_stacking_e21#538,improveit_category_e22#539,improveit_keyword_e23#540,omtr_vi_id_e24#541,aprimo_id_e25#542,link_name_e26#543,user_agent_e27#544,page_type_desc_e28#545,site_region_language_e29#546,social_campaigns_signup_e30#547,session_history_e31#548,external_data_source_type_e32#549,rmc_code_e33#550,video_name_e34#551,video_segment_e35#552,video_content_e36#553,free_user_prod_name_e37#554,web_voc_id_e38#555,ic_code_e39#556,hit_count_e40#557,signup_form_orig_page_e41#558,e42#559,e43#560,e44#561,pyament_plan_type_e45#562,purchased_product_name_e46#563,purchased_product_category_e47#564,link_loc_name_e48#565,link_pg_loc_name_e49#566,email_test_counter_e50#567,email_preview_counter_e51#568,email_send_counter_e52#569,time_emcf_tpicker_e53#570,time_emcf_editor_e54#571,time_emcf_scheduler_e55#572,time_emcf_build_e56#573,time_emcf_send_success_e57#574,timestamp_e58#575,templates_returned_e59#576,email_agent_id_e60#577,template_name_e61#578,template_hit_counter_e62#579,template_taxonomy_e63#580,email_home_taxonomy_e64#581,email_home_hit_counter_e65#582,editor_taxonomy_t66#583,editor_block_detail_e67#584,editor_block_title_e68#585,editor_hit_counter_e69#586,email_insert_detail_e70#587,email_creation_source_e71#588,e72#589,e73#590,e74#591,adlens_ef_id_e75#592,event_list#593,exclude_hit#594,first_hit_pagename#595,first_hit_page_url#596,first_hit_referrer#597,first_hit_time_gmt#598L,geo_city#599,geo_country#600,geo_dma#601,geo_region#602,geo_zip#603,hier1#604,hier2#605,hier3#606,hier4#607,hier5#608,hitid_high#609L,hitid_low#610L,hit_source#611,hit_time_gmt#612L,homepage#613,hourly_visitor#614,ip#615,ip2#616,java_enabled#617,javascript#618,j_jscript#619,language#620,last_hit_time_gmt#621L,last_purchase_num#622,last_purchase_time_gmt#623L,mobile_id#624L,monthly_visitor#625L,mvvar1#626,mvvar2#627,mvvar3#628,namespace#629,new_visit#630,os#631L,page_event#632,page_event_var1#633,page_event_var2#634,page_event_var3#635,page_name#636,page_type#637,page_url#638,paid_search#639,partner_plugins#640,persistent_cookie#641,plugins#642,post_browser_height#643,post_browser_width#644,post_campaign#645,post_channel#646,post_cookies#647,post_currency#648,post_cust_hit_time_gmt#649L,post_cust_visid#650,soid_pe1#651,lpuid_pe2#652,customer_status_pe3#653,partner_uid_pe4#654,ab_split_group_pe5#655,application_pe6#656,page_name_pe7#657,pe8#658,pe9#659,pe10#660,pe11#661,pe12#662,pe13#663,pe14#664,pe15#665,pe16#666,pe17#667,pe18#668,pe19#669,pe20#670,campaign_stacking_pe21#671,category_name_pe22#672,improvedit_keyword_pe23#673,omtr_vi_id_pe24#674,aprimo_id_pe25#675,link_name_pe26#676,pe27#677,pe28#678,site_region_language_pe29#679,pe30#680,session_history_pe31#681,pe32#682,rmc_code_pe33#683,video_name_pe34#684,pe35#685,pe36#686,pe37#687,pe38#688,ic_code_pe39#689,template_hit_crt_pe40#690,pe41#691,pe42#692,pe43#693,pe44#694,payment_plan_type_pe45#695,purchased_product_name_pe46#696,purchased_product_category_pe47#697,link_loc_name_pe48#698,link_pg_loc_name_pe49#699,email_test_ctr_pe50#700,email_preview_ctr_pe51#701,email_send_ctr_pe52#702,time_emcf_tpicker_pe53#703,time_emcf_editor_pe54#704,time_emcf_scheduler_pe55#705,time_emcf_build_pe56#706,time_emcf_send_success_pe57#707,pe58#708,pe59#709,email_agent_id_pe60#710,template_name_pe61#711,template_hit_ctr_pe62#712,template_taxonomy_pe63#713,email_home_taxonomy_pe64#714,email_home_hit_crt_pe65#715,editor_taxonomy_pe66#716,editor_block_detail_pe67#717,editor_block_title_pe68#718,editor_hit_crt_pe69#719,editor_insert_detail_pe70#720,email_creation_source_pe71#721,pe72#722,pe73#723,pe74#724,adlens_ef_id_pe75#725,post_event_list#726,post_hier1#727,post_hier2#728,post_hier3#729,post_hier4#730,post_hier5#731,post_java_enabled#732,post_keywords#733,post_mvvar1#734,post_mvvar2#735,post_mvvar3#736,post_page_event#737,post_page_event_var1#738,post_page_event_var2#739,post_page_event_var3#740,post_pagename#741,post_pagename_no_url#742,post_page_type#743,post_page_url#744,post_partner_plugins#745,post_persistent_cookie#746,post_product_list#747,soid_pp1#748,lpuid_pp2#749,custom_status_pp3#750,internal_external_pp4#751,ab_split_group_pp5#752,application_pp6#753,internal_search_query_pp7#754,pp8#755,pp9#756,pp10#757,pp11#758,sops_pp12#759,pp13#760,pp14#761,pp15#762,pp16#763,pp17#764,pp18#765,signup_form_pp19#766,pp20#767,pp21#768,category_name_pp22#769,improveit_keyword_pp23#770,pp24#771,sops_hero_name_pp25#772,video_name_pp26#773,pp27#774,topnav_link_name_pp28#775,pp29#776,pp30#777,pp31#778,pp32#779,pp33#780,pp34#781,pp35#782,pp36#783,pp37#784,pp38#785,pp39#786,link_name_pp40#787,link_loc_name_pp41#788,link_pg_loc_name_pp42#789,pp43#790,pp44#791,pp45#792,pp46#793,pp47#794,pp48#795,template_taxonomy_pp49#796,template_name_pp50#797,email_home_taxonomy_pp51#798,editor_taxonomy_pp52#799,kb_details_pp53#800,pp54#801,pp55#802,pp56#803,pp57#804,pp58#805,pp59#806,pp60#807,pp61#808,pp62#809,pp63#810,pp64#811,pp65#812,pp66#813,pp67#814,pp68#815,pp69#816,pp70#817,pp71#818,pp72#819,pp73#820,pp74#821,pp75#822,post_purchaseid#823,post_referrer#824,post_search_engine#825,post_state#826,post_survey#827,post_tnt#828,post_transactionid#829,post_t_time_info#830,post_visid_high#831L,post_visid_low#832L,post_visid_type#833,post_zip#834,p_plugins#835,prev_page#836L,product_list#837,product_merchandising#838,soid_p1#839,p2#840,p3#841,p4#842,p5#843,p6#844,p7#845,p8#846,p9#847,p10#848,p11#849,p12#850,p13#851,p14#852,p15#853,p16#854,p17#855,p18#856,signup_form_p19#857,p20#858,p21#859,improveit_category_p22#860,improveit_keyword_p23#861,test_drive_type_p24#862,sops_hero_name_p25#863,video_name_p26#864,previous_page_name_p27#865,topnav_link_name_p28#866,p29#867,marketplace_detail_p30#868,p31#869,p32#870,p33#871,p34#872,p35#873,p36#874,p37#875,p38#876,p39#877,link_name_p40#878,link_loc_name_p41#879,link_pg_loc_name_p42#880,p43#881,p44#882,p45#883,p46#884,p47#885,timestamp_p48#886,template_taxonomy_p49#887,template_name_p50#888,email_home_taxonomy_p51#889,editor_taxonomy_p52#890,kb_detail_p53#891,stock_image_name_p54#892,sent_template_name_p55#893,p56#894,p57#895,p58#896,p59#897,p60#898,p61#899,p62#900,p63#901,p64#902,p65#903,p66#904,p67#905,p68#906,p69#907,p70#908,p71#909,p72#910,p73#911,p74#912,p75#913,purchaseid#914,quarterly_visitor#915L,ref_domain#916,referrer#917,ref_type#918,resolution#919,sampled_hit#920,search_engine#921,search_page_num#922L,secondary_hit#923,service#924,sourceid#925L,s_resolution#926,state#927,stats_server#928,tnt#929,tnt_post_vista#930,transactionid#931,truncated_hit#932,t_time_info#933,ua_color#934,ua_os#935,ua_pixels#936,user_agent#937,user_hash#938L,userid#939L,username#940,user_server#941,va_closer_detail#942,va_closer_id#943L,va_finder_detail#944,va_finder_id#945L,va_instance_event#946,va_new_engagement#947,visid_high#948L,visid_low#949L,visid_new#950,visid_timestamp#951L,visid_type#952,visit_keywords#953,visit_num#954,visit_page_num#955,visit_referrer#956,visit_search_engine#957,visit_start_pagename#958,visit_start_page_url#959,visit_start_time_gmt#960L,weekly_visitor#961,yearly_visitor#962L,zip#963,browser_desc#964,color_desc#965,connection_type_desc#966,country_desc#967,event_list_desc#968,javascript_desc#969,language_desc#970,os_desc#971,plugins_desc#972,ref_type_desc#973,resolution_desc#974,search_engine_desc#975]  ConvertToSafe   TungstenProject [soid_e1#30,visid_high#460L,visid_low#461L,date_time#25L]    Filter (instr(event_list#105,202) > 0)     Scan ParquetRelation[hdfs://xxx/data/event_parquet/2015/08/21][accept_language#0,browser#1L,browser_height#2,browser_width#3,campaign#4,c_color#5,channel#6,click_action#7,click_action_type#8,click_context#9,click_context_type#10,click_sourceId#11L,click_tag#12,code_ver#13,color#14,connection_type#15,cookies#16,country#17,ct_connect_type#18,currency#19,curr_factor#20,curr_rate#21,cust_hit_time_gmt#22L,cust_visid#23,daily_visitor#24,date_time#25L,domain#26,duplicated_from#27,duplicate_events#28,duplicate_purchase#29,soid_e1#30,lpuid_e2#31,customer_status_e3#32,partner_uid_e4#33,ab_split_group_e5#34,application_e6#35,page_name_e7#36,originating_page_e8#37,email_signup_e9#38,survey_signup_e10#39,event_signup_e11#40,email_paying_e12#41,survey_paying_e13#42,event_paying_e14#43,image_hosting_paying_e15#44,archive_paying_e16#45,internal_search_query_e17#46,internal_search_page_e18#47,download_file_name_e19#48,download_file_page_e20#49,campaign_stacking_e21#50,improveit_category_e22#51,improveit_keyword_e23#52,omtr_vi_id_e24#53,aprimo_id_e25#54,link_name_e26#55,user_agent_e27#56,page_type_desc_e28#57,site_region_language_e29#58,social_campaigns_signup_e30#59,session_history_e31#60,external_data_source_type_e32#61,rmc_code_e33#62,video_name_e34#63,video_segment_e35#64,video_content_e36#65,free_user_prod_name_e37#66,web_voc_id_e38#67,ic_code_e39#68,hit_count_e40#69,signup_form_orig_page_e41#70,e42#71,e43#72,e44#73,pyament_plan_type_e45#74,purchased_product_name_e46#75,purchased_product_category_e47#76,link_loc_name_e48#77,link_pg_loc_name_e49#78,email_test_counter_e50#79,email_preview_counter_e51#80,email_send_counter_e52#81,time_emcf_tpicker_e53#82,time_emcf_editor_e54#83,time_emcf_scheduler_e55#84,time_emcf_build_e56#85,time_emcf_send_success_e57#86,timestamp_e58#87,templates_returned_e59#88,email_agent_id_e60#89,template_name_e61#90,template_hit_counter_e62#91,template_taxonomy_e63#92,email_home_taxonomy_e64#93,email_home_hit_counter_e65#94,editor_taxonomy_t66#95,editor_block_detail_e67#96,editor_block_title_e68#97,editor_hit_counter_e69#98,email_insert_detail_e70#99,email_creation_source_e71#100,e72#101,e73#102,e74#103,adlens_ef_id_e75#104,event_list#105,exclude_hit#106,first_hit_pagename#107,first_hit_page_url#108,first_hit_referrer#109,first_hit_time_gmt#110L,geo_city#111,geo_country#112,geo_dma#113,geo_region#114,geo_zip#115,hier1#116,hier2#117,hier3#118,hier4#119,hier5#120,hitid_high#121L,hitid_low#122L,hit_source#123,hit_time_gmt#124L,homepage#125,hourly_visitor#126,ip#127,ip2#128,java_enabled#129,javascript#130,j_jscript#131,language#132,last_hit_time_gmt#133L,last_purchase_num#134,last_purchase_time_gmt#135L,mobile_id#136L,monthly_visitor#137L,mvvar1#138,mvvar2#139,mvvar3#140,namespace#141,new_visit#142,os#143L,page_event#144,page_event_var1#145,page_event_var2#146,page_event_var3#147,page_name#148,page_type#149,page_url#150,paid_search#151,partner_plugins#152,persistent_cookie#153,plugins#154,post_browser_height#155,post_browser_width#156,post_campaign#157,post_channel#158,post_cookies#159,post_currency#160,post_cust_hit_time_gmt#161L,post_cust_visid#162,soid_pe1#163,lpuid_pe2#164,customer_status_pe3#165,partner_uid_pe4#166,ab_split_group_pe5#167,application_pe6#168,page_name_pe7#169,pe8#170,pe9#171,pe10#172,pe11#173,pe12#174,pe13#175,pe14#176,pe15#177,pe16#178,pe17#179,pe18#180,pe19#181,pe20#182,campaign_stacking_pe21#183,category_name_pe22#184,improvedit_keyword_pe23#185,omtr_vi_id_pe24#186,aprimo_id_pe25#187,link_name_pe26#188,pe27#189,pe28#190,site_region_language_pe29#191,pe30#192,session_history_pe31#193,pe32#194,rmc_code_pe33#195,video_name_pe34#196,pe35#197,pe36#198,pe37#199,pe38#200,ic_code_pe39#201,template_hit_crt_pe40#202,pe41#203,pe42#204,pe43#205,pe44#206,payment_plan_type_pe45#207,purchased_product_name_pe46#208,purchased_product_category_pe47#209,link_loc_name_pe48#210,link_pg_loc_name_pe49#211,email_test_ctr_pe50#212,email_preview_ctr_pe51#213,email_send_ctr_pe52#214,time_emcf_tpicker_pe53#215,time_emcf_editor_pe54#216,time_emcf_scheduler_pe55#217,time_emcf_build_pe56#218,time_emcf_send_success_pe57#219,pe58#220,pe59#221,email_agent_id_pe60#222,template_name_pe61#223,template_hit_ctr_pe62#224,template_taxonomy_pe63#225,email_home_taxonomy_pe64#226,email_home_hit_crt_pe65#227,editor_taxonomy_pe66#228,editor_block_detail_pe67#229,editor_block_title_pe68#230,editor_hit_crt_pe69#231,editor_insert_detail_pe70#232,email_creation_source_pe71#233,pe72#234,pe73#235,pe74#236,adlens_ef_id_pe75#237,post_event_list#238,post_hier1#239,post_hier2#240,post_hier3#241,post_hier4#242,post_hier5#243,post_java_enabled#244,post_keywords#245,post_mvvar1#246,post_mvvar2#247,post_mvvar3#248,post_page_event#249,post_page_event_var1#250,post_page_event_var2#251,post_page_event_var3#252,post_pagename#253,post_pagename_no_url#254,post_page_type#255,post_page_url#256,post_partner_plugins#257,post_persistent_cookie#258,post_product_list#259,soid_pp1#260,lpuid_pp2#261,custom_status_pp3#262,internal_external_pp4#263,ab_split_group_pp5#264,application_pp6#265,internal_search_query_pp7#266,pp8#267,pp9#268,pp10#269,pp11#270,sops_pp12#271,pp13#272,pp14#273,pp15#274,pp16#275,pp17#276,pp18#277,signup_form_pp19#278,pp20#279,pp21#280,category_name_pp22#281,improveit_keyword_pp23#282,pp24#283,sops_hero_name_pp25#284,video_name_pp26#285,pp27#286,topnav_link_name_pp28#287,pp29#288,pp30#289,pp31#290,pp32#291,pp33#292,pp34#293,pp35#294,pp36#295,pp37#296,pp38#297,pp39#298,link_name_pp40#299,link_loc_name_pp41#300,link_pg_loc_name_pp42#301,pp43#302,pp44#303,pp45#304,pp46#305,pp47#306,pp48#307,template_taxonomy_pp49#308,template_name_pp50#309,email_home_taxonomy_pp51#310,editor_taxonomy_pp52#311,kb_details_pp53#312,pp54#313,pp55#314,pp56#315,pp57#316,pp58#317,pp59#318,pp60#319,pp61#320,pp62#321,pp63#322,pp64#323,pp65#324,pp66#325,pp67#326,pp68#327,pp69#328,pp70#329,pp71#330,pp72#331,pp73#332,pp74#333,pp75#334,post_purchaseid#335,post_referrer#336,post_search_engine#337,post_state#338,post_survey#339,post_tnt#340,post_transactionid#341,post_t_time_info#342,post_visid_high#343L,post_visid_low#344L,post_visid_type#345,post_zip#346,p_plugins#347,prev_page#348L,product_list#349,product_merchandising#350,soid_p1#351,p2#352,p3#353,p4#354,p5#355,p6#356,p7#357,p8#358,p9#359,p10#360,p11#361,p12#362,p13#363,p14#364,p15#365,p16#366,p17#367,p18#368,signup_form_p19#369,p20#370,p21#371,improveit_category_p22#372,improveit_keyword_p23#373,test_drive_type_p24#374,sops_hero_name_p25#375,video_name_p26#376,previous_page_name_p27#377,topnav_link_name_p28#378,p29#379,marketplace_detail_p30#380,p31#381,p32#382,p33#383,p34#384,p35#385,p36#386,p37#387,p38#388,p39#389,link_name_p40#390,link_loc_name_p41#391,link_pg_loc_name_p42#392,p43#393,p44#394,p45#395,p46#396,p47#397,timestamp_p48#398,template_taxonomy_p49#399,template_name_p50#400,email_home_taxonomy_p51#401,editor_taxonomy_p52#402,kb_detail_p53#403,stock_image_name_p54#404,sent_template_name_p55#405,p56#406,p57#407,p58#408,p59#409,p60#410,p61#411,p62#412,p63#413,p64#414,p65#415,p66#416,p67#417,p68#418,p69#419,p70#420,p71#421,p72#422,p73#423,p74#424,p75#425,purchaseid#426,quarterly_visitor#427L,ref_domain#428,referrer#429,ref_type#430,resolution#431,sampled_hit#432,search_engine#433,search_page_num#434L,secondary_hit#435,service#436,sourceid#437L,s_resolution#438,state#439,stats_server#440,tnt#441,tnt_post_vista#442,transactionid#443,truncated_hit#444,t_time_info#445,ua_color#446,ua_os#447,ua_pixels#448,user_agent#449,user_hash#450L,userid#451L,username#452,user_server#453,va_closer_detail#454,va_closer_id#455L,va_finder_detail#456,va_finder_id#457L,va_instance_event#458,va_new_engagement#459,visid_high#460L,visid_low#461L,visid_new#462,visid_timestamp#463L,visid_type#464,visit_keywords#465,visit_num#466,visit_page_num#467,visit_referrer#468,visit_search_engine#469,visit_start_pagename#470,visit_start_page_url#471,visit_start_time_gmt#472L,weekly_visitor#473,yearly_visitor#474L,zip#475,browser_desc#476,color_desc#477,connection_type_desc#478,country_desc#479,event_list_desc#480,javascript_desc#481,language_desc#482,os_desc#483,plugins_desc#484,ref_type_desc#485,resolution_desc#486,search_engine_desc#487]