[jira] [Created] (ORC-547) ORC write on Map Reduce fwk is extremely slow

["Thrinath Dosapati (Jira)" <ji...@apache.org>]

Thrinath Dosapati created ORC-547:
-------------------------------------

             Summary: ORC write on Map Reduce fwk is extremely slow
                 Key: ORC-547
                 URL: https://issues.apache.org/jira/browse/ORC-547
             Project: ORC
          Issue Type: Test
          Components: MapReduce
    Affects Versions: 1.3.3
         Environment: Map Reduce FWK
            Reporter: Thrinath Dosapati
         Attachments: orc_slow_write_log.txt, sample_record.json

Recently, we have encountered cases where the ORC write is extremely slow for certain workloads. 

What could be the reason for the slowness?

Schema : 

struct<rc:struct<cc:struct<appv:string,cht:string>,pc,ac:array<struct<layer:string,abid:string>>,mp:string,rsc:bigint,pt:string,ai:struct<supercat:string,subcat:string,v:string,cat:string>,prid:string,pid:array<string>,rid:string,uc:struct<abid:string,aid:string>,p:array<struct<productid:string,meta:array<struct<mv:string,mk:string>>,nid:string,lid:string>>,sc:array<struct<score:double,sid:string>>,ui:struct<ss:string,dg:struct<mds:string,fds:string>,ps:string,bg:struct<ms:string,fs:string>,ms:string,ul:array<struct<c:string,s:string,p:string>>,iscc:boolean,ic:boolean,rfmb:struct<rb:string,fb:string,mb:string,rfmsg:string,imlb:boolean>>,pck:string,pi:string,dc:struct<os:string,ip:string,did:string>>,rws:array<struct<rccs:array<struct<eid:string,bc:string,mp:string,lid:string,nid:string,cm:array<struct<mv:string,mk:string>>,mtomlfs:array<struct<rv:string,lid:string,ms:string,mid:string,mv:string,mlfs:array<struct<fw:string,f:string>>>>,rpid:string,et:string,dt:string,cs:string,ct:string,t:string,cid:string>>,wm:array<struct<mv:string,mk:string>>,mtomlfs:array<struct<rv:string,lid:string,ms:string,mid:string,mv:string,mlfs:array<struct<fw:string,f:string>>>>,wc:struct<murl:string,rt:string,djct:string,wimpid:string,va:string,title:string,ws:string,wc:string,mtext:string,wt:string,vt:string,urms:array<struct<rk:string,dc:bigint>>,mrcc:bigint,sc:bigint>>>>

 

Logs and sample records are attached for reference.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)