You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by mo...@open-insights.co.in on 2017/10/25 07:29:21 UTC
Auto Partitioning in PuthiveStreming Processor
Hi All,
I am trying to write data to dynamic partitions in Hive using
PutHiveStreaming Processor.
It fails in case of auto partitioning. Though it is working fine if
partition is already created.
Nifi throws the following error in case of auto partitioning:-
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Failed to create
HiveWriter for endpoint: {metaStoreUri='thrift://localhost:9083',
database='test', table='jsondpi3', partitionVals=[356945013] }:
org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
table='jsondpi3', partitionVals=[356945013] }
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Error connecting
to Hive endpoint: table jsondpi3 at thrift://localhost:9083
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Hive Streaming
connect/write error, flow file will be penalized and routed to retry.
org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
table='jsondpi3', partitionVals=[356945013] }:
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive
Streaming connect/write error, flow file will be penalized and routed to
retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed
connecting to EndPoint {metaStoreUri='thrift://localhost:9083',
database='test', table='jsondpi3', partitionVals=[356945013] }
Nifi app log shows the following log :
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive
Streaming connect/write error, flow file will be penalized and routed to
retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed
connecting to EndPoint {metaStoreUri='thrift://localhost:9083',
database='test', table='jsondpi3', partitionVals=[356945013] }
at
org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1
(PutHiveStreaming.java:527)
at
org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andTh
en$0(ExceptionHandler.java:54)
at
org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(
PutHiveStreaming.java:545)
at
org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHan
dler.java:148)
at
org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHive
Streaming.java:677)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardPr
ocessSession.java:2174)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardPr
ocessSession.java:2144)
at
org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.
java:631)
at
org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveS
treaming.java:555)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFun
ctions.java:114)
at
org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackO
nFailure.java:184)
at
org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.
java:555)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessor
Node.java:1119)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(Continuall
yRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(Continuall
yRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(Timer
DrivenSchedulingAgent.java:128)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
49)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
24)
at java.lang.Thread.run(Thread.java:748)
Error in detail is attached.
Property details Is also attached.
Please let me know what I am doing wrong.
Regards,
Mohit Jain
RE: Auto Partitioning in PuthiveStreming Processor
Posted by mo...@open-insights.co.in.
Answers to your questions –
Do you have logs on Hive's side? Could it be a permission issue (the user not being allowed to create a new directory for the new partition)?
I don’t have logs on the Hive Side. Earlier I thought it to be a permission issue, but both Nifi and Hive have a same user.
Can you display the content of a flow file going in the processor (with the avro content viewer)? Is the value 356945013 the expected partition value?
Avro files that are input to the PutHiveStreaming processor is a valid avro.
Yes, 356945013 is the partition value.
Thanks.
From: Pierre Villard [mailto:pierre.villard.fr@gmail.com]
Sent: 25 October 2017 15:46
To: users@nifi.apache.org
Subject: Re: Auto Partitioning in PuthiveStreming Processor
Few additional questions (I don't have any specific explanation to give you but I think I remember using it successfully in the past):
Do you have logs on Hive's side? Could it be a permission issue (the user not being allowed to create a new directory for the new partition)?
Can you display the content of a flow file going in the processor (with the avro content viewer)? Is the value 356945013 the expected partition value?
Pierre
2017-10-25 12:06 GMT+02:00 <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> >:
Hi,
JSONDPI3 is an external table. Though I am facing the same issue in both the managed and external table. Sharing with you the details of the JSONDPI3.
Thanks.
From: Pierre Villard [mailto:pierre.villard.fr@gmail.com <ma...@gmail.com> ]
Sent: 25 October 2017 15:30
To: users@nifi.apache.org <ma...@nifi.apache.org>
Subject: Re: Auto Partitioning in PuthiveStreming Processor
In your stack trace / configuration, I see that the table name is jsondpi3, but the table in your last message is different. Are the descriptions of both tables the same?
Pierre
2017-10-25 11:50 GMT+02:00 <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> >:
Hi Pierre,
Pleae find the desc formatted :
Regards,
Mohit
From: Pierre Villard [mailto:pierre.villard.fr@gmail.com <ma...@gmail.com> ]
Sent: 25 October 2017 15:14
To: users@nifi.apache.org <ma...@nifi.apache.org>
Subject: Re: Auto Partitioning in PuthiveStreming Processor
Hi Mohit,
Can you share a 'describe formatted' of your table in Hive?
Thanks!
2017-10-25 9:29 GMT+02:00 <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> >:
Hi All,
I am trying to write data to dynamic partitions in Hive using PutHiveStreaming Processor.
It fails in case of auto partitioning. Though it is working fine if partition is already created.
Nifi throws the following error in case of auto partitioning:-
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Error connecting to Hive endpoint: table jsondpi3 at thrift://localhost:9083
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
Nifi app log shows the following log :
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:527)
at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:545)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHiveStreaming.java:677)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2174)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2144)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:631)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:555)
at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:555)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Error in detail is attached.
Property details Is also attached.
Please let me know what I am doing wrong.
Regards,
Mohit Jain
Re: Auto Partitioning in PuthiveStreming Processor
Posted by Pierre Villard <pi...@gmail.com>.
Few additional questions (I don't have any specific explanation to give you
but I think I remember using it successfully in the past):
Do you have logs on Hive's side? Could it be a permission issue (the user
not being allowed to create a new directory for the new partition)?
Can you display the content of a flow file going in the processor (with the
avro content viewer)? Is the value 356945013 the expected partition value?
Pierre
2017-10-25 12:06 GMT+02:00 <mo...@open-insights.co.in>:
> Hi,
>
> JSONDPI3 is an external table. Though I am facing the same issue in both
> the managed and external table. Sharing with you the details of the
> JSONDPI3.
>
>
>
> Thanks.
>
> *From:* Pierre Villard [mailto:pierre.villard.fr@gmail.com]
> *Sent:* 25 October 2017 15:30
>
> *To:* users@nifi.apache.org
> *Subject:* Re: Auto Partitioning in PuthiveStreming Processor
>
>
>
> In your stack trace / configuration, I see that the table name is
> jsondpi3, but the table in your last message is different. Are the
> descriptions of both tables the same?
>
> Pierre
>
>
>
> 2017-10-25 11:50 GMT+02:00 <mo...@open-insights.co.in>:
>
> Hi Pierre,
>
>
>
> Pleae find the desc formatted :
>
>
>
>
>
> Regards,
>
> Mohit
>
> *From:* Pierre Villard [mailto:pierre.villard.fr@gmail.com]
> *Sent:* 25 October 2017 15:14
> *To:* users@nifi.apache.org
> *Subject:* Re: Auto Partitioning in PuthiveStreming Processor
>
>
>
> Hi Mohit,
>
> Can you share a 'describe formatted' of your table in Hive?
>
> Thanks!
>
>
>
> 2017-10-25 9:29 GMT+02:00 <mo...@open-insights.co.in>:
>
> Hi All,
>
>
>
> I am trying to write data to dynamic partitions in Hive using
> PutHiveStreaming Processor.
>
> It fails in case of auto partitioning. Though it is working fine if
> partition is already created.
>
>
>
> Nifi throws the following error in case of auto partitioning:-
>
>
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Failed to
> create HiveWriter for endpoint: {metaStoreUri='thrift://localhost:9083',
> database='test', table='jsondpi3', partitionVals=[356945013] }:
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Error
> connecting to Hive endpoint: table jsondpi3 at thrift://localhost:9083
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Hive Streaming
> connect/write error, flow file will be penalized and routed to retry.
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.processors.
> hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write
> error, flow file will be penalized and routed to retry.
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }
>
>
>
> Nifi app log shows the following log :
>
>
>
> org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException:
> Hive Streaming connect/write error, flow file will be penalized and routed
> to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed
> connecting to EndPoint {metaStoreUri='thrift://localhost:9083',
> database='test', table='jsondpi3', partitionVals=[356945013] }
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onHiveRecordsError$1(PutHiveStreaming.java:527)
>
> at org.apache.nifi.processor.util.pattern.ExceptionHandler$
> OnError.lambda$andThen$0(ExceptionHandler.java:54)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onHiveRecordError$2(PutHiveStreaming.java:545)
>
> at org.apache.nifi.processor.util.pattern.ExceptionHandler.
> execute(ExceptionHandler.java:148)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onTrigger$12(PutHiveStreaming.java:677)
>
> at org.apache.nifi.controller.repository.
> StandardProcessSession.read(StandardProcessSession.java:2174)
>
> at org.apache.nifi.controller.repository.
> StandardProcessSession.read(StandardProcessSession.java:2144)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.
> onTrigger(PutHiveStreaming.java:631)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onTrigger$4(PutHiveStreaming.java:555)
>
> at org.apache.nifi.processor.util.pattern.PartialFunctions.
> onTrigger(PartialFunctions.java:114)
>
> at org.apache.nifi.processor.util.pattern.
> RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.
> onTrigger(PutHiveStreaming.java:555)
>
> at org.apache.nifi.controller.StandardProcessorNode.
> onTrigger(StandardProcessorNode.java:1119)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
>
> Error in detail is attached.
>
> Property details Is also attached.
>
>
>
> Please let me know what I am doing wrong.
>
>
>
> Regards,
>
> Mohit Jain
>
>
>
>
>
RE: Auto Partitioning in PuthiveStreming Processor
Posted by mo...@open-insights.co.in.
Hi,
JSONDPI3 is an external table. Though I am facing the same issue in both the managed and external table. Sharing with you the details of the JSONDPI3.
Thanks.
From: Pierre Villard [mailto:pierre.villard.fr@gmail.com]
Sent: 25 October 2017 15:30
To: users@nifi.apache.org
Subject: Re: Auto Partitioning in PuthiveStreming Processor
In your stack trace / configuration, I see that the table name is jsondpi3, but the table in your last message is different. Are the descriptions of both tables the same?
Pierre
2017-10-25 11:50 GMT+02:00 <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> >:
Hi Pierre,
Pleae find the desc formatted :
Regards,
Mohit
From: Pierre Villard [mailto:pierre.villard.fr@gmail.com <ma...@gmail.com> ]
Sent: 25 October 2017 15:14
To: users@nifi.apache.org <ma...@nifi.apache.org>
Subject: Re: Auto Partitioning in PuthiveStreming Processor
Hi Mohit,
Can you share a 'describe formatted' of your table in Hive?
Thanks!
2017-10-25 9:29 GMT+02:00 <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> >:
Hi All,
I am trying to write data to dynamic partitions in Hive using PutHiveStreaming Processor.
It fails in case of auto partitioning. Though it is working fine if partition is already created.
Nifi throws the following error in case of auto partitioning:-
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Error connecting to Hive endpoint: table jsondpi3 at thrift://localhost:9083
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
Nifi app log shows the following log :
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:527)
at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:545)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHiveStreaming.java:677)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2174)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2144)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:631)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:555)
at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:555)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Error in detail is attached.
Property details Is also attached.
Please let me know what I am doing wrong.
Regards,
Mohit Jain
Re: Auto Partitioning in PuthiveStreming Processor
Posted by Pierre Villard <pi...@gmail.com>.
In your stack trace / configuration, I see that the table name is jsondpi3,
but the table in your last message is different. Are the descriptions of
both tables the same?
Pierre
2017-10-25 11:50 GMT+02:00 <mo...@open-insights.co.in>:
> Hi Pierre,
>
>
>
> Pleae find the desc formatted :
>
>
>
>
>
> Regards,
>
> Mohit
>
> *From:* Pierre Villard [mailto:pierre.villard.fr@gmail.com]
> *Sent:* 25 October 2017 15:14
> *To:* users@nifi.apache.org
> *Subject:* Re: Auto Partitioning in PuthiveStreming Processor
>
>
>
> Hi Mohit,
>
> Can you share a 'describe formatted' of your table in Hive?
>
> Thanks!
>
>
>
> 2017-10-25 9:29 GMT+02:00 <mo...@open-insights.co.in>:
>
> Hi All,
>
>
>
> I am trying to write data to dynamic partitions in Hive using
> PutHiveStreaming Processor.
>
> It fails in case of auto partitioning. Though it is working fine if
> partition is already created.
>
>
>
> Nifi throws the following error in case of auto partitioning:-
>
>
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Failed to
> create HiveWriter for endpoint: {metaStoreUri='thrift://localhost:9083',
> database='test', table='jsondpi3', partitionVals=[356945013] }:
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Error
> connecting to Hive endpoint: table jsondpi3 at thrift://localhost:9083
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Hive Streaming
> connect/write error, flow file will be penalized and routed to retry.
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.processors.
> hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write
> error, flow file will be penalized and routed to retry.
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }
>
>
>
> Nifi app log shows the following log :
>
>
>
> org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException:
> Hive Streaming connect/write error, flow file will be penalized and routed
> to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed
> connecting to EndPoint {metaStoreUri='thrift://localhost:9083',
> database='test', table='jsondpi3', partitionVals=[356945013] }
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onHiveRecordsError$1(PutHiveStreaming.java:527)
>
> at org.apache.nifi.processor.util.pattern.ExceptionHandler$
> OnError.lambda$andThen$0(ExceptionHandler.java:54)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onHiveRecordError$2(PutHiveStreaming.java:545)
>
> at org.apache.nifi.processor.util.pattern.ExceptionHandler.
> execute(ExceptionHandler.java:148)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onTrigger$12(PutHiveStreaming.java:677)
>
> at org.apache.nifi.controller.repository.
> StandardProcessSession.read(StandardProcessSession.java:2174)
>
> at org.apache.nifi.controller.repository.
> StandardProcessSession.read(StandardProcessSession.java:2144)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.
> onTrigger(PutHiveStreaming.java:631)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onTrigger$4(PutHiveStreaming.java:555)
>
> at org.apache.nifi.processor.util.pattern.PartialFunctions.
> onTrigger(PartialFunctions.java:114)
>
> at org.apache.nifi.processor.util.pattern.
> RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.
> onTrigger(PutHiveStreaming.java:555)
>
> at org.apache.nifi.controller.StandardProcessorNode.
> onTrigger(StandardProcessorNode.java:1119)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
>
> Error in detail is attached.
>
> Property details Is also attached.
>
>
>
> Please let me know what I am doing wrong.
>
>
>
> Regards,
>
> Mohit Jain
>
>
>
RE: Auto Partitioning in PuthiveStreming Processor
Posted by mo...@open-insights.co.in.
Hi Pierre,
Pleae find the desc formatted :
Regards,
Mohit
From: Pierre Villard [mailto:pierre.villard.fr@gmail.com]
Sent: 25 October 2017 15:14
To: users@nifi.apache.org
Subject: Re: Auto Partitioning in PuthiveStreming Processor
Hi Mohit,
Can you share a 'describe formatted' of your table in Hive?
Thanks!
2017-10-25 9:29 GMT+02:00 <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> >:
Hi All,
I am trying to write data to dynamic partitions in Hive using PutHiveStreaming Processor.
It fails in case of auto partitioning. Though it is working fine if partition is already created.
Nifi throws the following error in case of auto partitioning:-
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Error connecting to Hive endpoint: table jsondpi3 at thrift://localhost:9083
12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
Nifi app log shows the following log :
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='test', table='jsondpi3', partitionVals=[356945013] }
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:527)
at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:545)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHiveStreaming.java:677)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2174)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2144)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:631)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:555)
at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:555)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Error in detail is attached.
Property details Is also attached.
Please let me know what I am doing wrong.
Regards,
Mohit Jain
Re: Auto Partitioning in PuthiveStreming Processor
Posted by Pierre Villard <pi...@gmail.com>.
Hi Mohit,
Can you share a 'describe formatted' of your table in Hive?
Thanks!
2017-10-25 9:29 GMT+02:00 <mo...@open-insights.co.in>:
> Hi All,
>
>
>
> I am trying to write data to dynamic partitions in Hive using
> PutHiveStreaming Processor.
>
> It fails in case of auto partitioning. Though it is working fine if
> partition is already created.
>
>
>
> Nifi throws the following error in case of auto partitioning:-
>
>
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Failed to
> create HiveWriter for endpoint: {metaStoreUri='thrift://localhost:9083',
> database='test', table='jsondpi3', partitionVals=[356945013] }:
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Error
> connecting to Hive endpoint: table jsondpi3 at thrift://localhost:9083
>
> 12:48:14 ISTERROR80ee3b56-68a5-13e1-3251-9cc44bebddf4
>
> PutHiveStreaming[id=80ee3b56-68a5-13e1-3251-9cc44bebddf4] Hive Streaming
> connect/write error, flow file will be penalized and routed to retry.
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }: org.apache.nifi.processors.
> hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write
> error, flow file will be penalized and routed to retry.
> org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to
> EndPoint {metaStoreUri='thrift://localhost:9083', database='test',
> table='jsondpi3', partitionVals=[356945013] }
>
>
>
> Nifi app log shows the following log :
>
>
>
> org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException:
> Hive Streaming connect/write error, flow file will be penalized and routed
> to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed
> connecting to EndPoint {metaStoreUri='thrift://localhost:9083',
> database='test', table='jsondpi3', partitionVals=[356945013] }
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onHiveRecordsError$1(PutHiveStreaming.java:527)
>
> at org.apache.nifi.processor.util.pattern.ExceptionHandler$
> OnError.lambda$andThen$0(ExceptionHandler.java:54)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onHiveRecordError$2(PutHiveStreaming.java:545)
>
> at org.apache.nifi.processor.util.pattern.ExceptionHandler.
> execute(ExceptionHandler.java:148)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onTrigger$12(PutHiveStreaming.java:677)
>
> at org.apache.nifi.controller.repository.
> StandardProcessSession.read(StandardProcessSession.java:2174)
>
> at org.apache.nifi.controller.repository.
> StandardProcessSession.read(StandardProcessSession.java:2144)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.
> onTrigger(PutHiveStreaming.java:631)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$
> onTrigger$4(PutHiveStreaming.java:555)
>
> at org.apache.nifi.processor.util.pattern.PartialFunctions.
> onTrigger(PartialFunctions.java:114)
>
> at org.apache.nifi.processor.util.pattern.
> RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
>
> at org.apache.nifi.processors.hive.PutHiveStreaming.
> onTrigger(PutHiveStreaming.java:555)
>
> at org.apache.nifi.controller.StandardProcessorNode.
> onTrigger(StandardProcessorNode.java:1119)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
>
> Error in detail is attached.
>
> Property details Is also attached.
>
>
>
> Please let me know what I am doing wrong.
>
>
>
> Regards,
>
> Mohit Jain
>