You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jorge Alpedrinha Ramos <ja...@gmail.com> on 2017/03/23 10:55:19 UTC
Missing region in S3Hook when using extra
Hi,
I've been using S3Hook for a while, but I've run into some issues for not
having a defined region. I can't really say why I can't access this bucket
without defining the region, but I know that there is no way to define
region in S3Hook via the connection extra parameter unless a config_file is
used, which is not always an available method.
I haven't found any ticket regarding this, is there any alternative ?
Regards,
Jorge
Re: Missing region in S3Hook when using extra
Posted by Robin Bartholdson <ro...@formulate.se>.
Well, actually the logic is to look for whether you have specified the host in the extra params, and if you haven’t — set host to ‘NoHostProvided’ (the default)
Great that you are willing to take a stab at that!
-Robin
> On 24 Mar 2017, at 11:16, Jorge Alpedrinha Ramos <ja...@gmail.com> wrote:
>
> Thank you Robin,
>
> Your code provides a simple solution for the problem, although I believe
> that forcing host to be defined may be unnecessary because most use cases
> won't require it. After looking at the host hack used with boto I really
> think S3Hook should start using boto3, the fact that you can't pass a
> region string such as *us-east-1* using host instead doesn't seem standard
> and is not reusable for any other purpose (other than s3).
>
> I think https://issues.apache.org/jira/browse/AIRFLOW-115 is the key point.
> I'll look into making a proposition on the matter.
>
> Thank you all for your help.
>
>
> On Thu, Mar 23, 2017 at 5:16 PM Gael Magnan <ga...@gmail.com> wrote:
>
>> Hi,
>>
>> we have had the same problem and basically did the same as Robin, the only
>> diff we have is we use the host from the connection instead of extra
>> params.
>>
>> I can't agree more with the statement that we need to really rethink the
>> S3Hook, and maybe make some basic S3/AWS operators.
>>
>> Gael
>>
>> Le jeu. 23 mars 2017 à 18:05, Robin Bartholdson <
>> robin.bartholdson@formulate.se> a écrit :
>>
>>> I submitted our patch in the mean time, do test it and see if it works
>> for
>>> you :-)
>>>
>>> https://github.com/apache/incubator-airflow/pull/2181
>>> https://issues.apache.org/jira/browse/AIRFLOW-1034
>>>
>>> -Robin
>>>
>>> On 23 Mar 2017, at 17:43, Robin Bartholdson <
>>> robin.bartholdson@formulate.se> wrote:
>>>
>>> There is some discussion in this JIRA,
>>> https://issues.apache.org/jira/browse/AIRFLOW-115, about moving to boto3
>>> and restructuring along the lines you discuss.
>>>
>>> I totally agree that it would indeed be very useful!
>>>
>>> -Robin
>>>
>>>
>>> On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <
>>> jalpedrinharamos@gmail.com> wrote:
>>>
>>> Hi Sumit,
>>>
>>> I was actually referring to what Robin (thank you) pointed out which is
>>> reason enough for having region in S3Hook.
>>>
>>> There are even other situations having the region would be beneficial,
>>> that's why the AwsHook has that definition, although I would agree that
>>> their goals are different they typically refer to the same credentials,
>>> just different use cases.
>>>
>>> This makes me feel that it becomes redundant to have both an S3
>> connection
>>> and an AWS connection with the same credentials just be able to access
>>> region. Wouldn't it be beneficial to implement S3Hook (and S3 connection)
>>> as a special case of AwsHook (and Aws connection) and be able to use both
>>> interchangeably ?
>>>
>>> I'm also aware that one uses boto and the other uses boto3, and it seems
>>> like the project is naturally slowly migrating towards the later. Maybe
>>> there is even some discussion regarding this that I'm not aware.
>>>
>>> If you think this is a desirable move, I may start to design this move,
>>> making sure nothing gets broken in the process.
>>>
>>> WDYT?
>>>
>>> On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson <
>>> robin.bartholdson@formulate.se> wrote:
>>>
>>> I _think_ that’s a different issue (do correct me if I’m wrong)
>>>
>>> For a number of regions (see the list on
>>> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html),
>>> you have to specify an extra host-parameter in the boto-S3Connection as
>>> they only support signature V4 and the host-param is required to have
>> this
>>> working (
>>>
>>>
>> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError
>>> )
>>>
>>> -Robin
>>>
>>> On 23 Mar 2017, at 15:10, Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>>
>>> Are we talking about this
>>> https://github.com/apache/incubator-airflow/pull/2176 ?
>>>
>>>
>>>
>>>
>>> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
>>> robin.bartholdson@formulate.se> wrote:
>>>
>>> Hi,
>>>
>>> We actually have a patch for this that allows you to specify the host
>>> parameter in the S3 config in the connections.
>>>
>>> Been thinking about submitting this for a while but have not had the
>> time,
>>> maybe now’s the time to do it :-)
>>>
>>> If you want to open a JIRA I can submit the patch we are using in
>>> production!
>>>
>>> -Robin
>>>
>>>
>>> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
>>> jalpedrinharamos@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I've been using S3Hook for a while, but I've run into some issues for not
>>> having a defined region. I can't really say why I can't access this
>> bucket
>>> without defining the region, but I know that there is no way to define
>>> region in S3Hook via the connection extra parameter unless a config_file
>> is
>>> used, which is not always an available method.
>>>
>>> I haven't found any ticket regarding this, is there any alternative ?
>>>
>>> Regards,
>>> Jorge
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
Re: Missing region in S3Hook when using extra
Posted by Jorge Alpedrinha Ramos <ja...@gmail.com>.
Thank you Robin,
Your code provides a simple solution for the problem, although I believe
that forcing host to be defined may be unnecessary because most use cases
won't require it. After looking at the host hack used with boto I really
think S3Hook should start using boto3, the fact that you can't pass a
region string such as *us-east-1* using host instead doesn't seem standard
and is not reusable for any other purpose (other than s3).
I think https://issues.apache.org/jira/browse/AIRFLOW-115 is the key point.
I'll look into making a proposition on the matter.
Thank you all for your help.
On Thu, Mar 23, 2017 at 5:16 PM Gael Magnan <ga...@gmail.com> wrote:
> Hi,
>
> we have had the same problem and basically did the same as Robin, the only
> diff we have is we use the host from the connection instead of extra
> params.
>
> I can't agree more with the statement that we need to really rethink the
> S3Hook, and maybe make some basic S3/AWS operators.
>
> Gael
>
> Le jeu. 23 mars 2017 à 18:05, Robin Bartholdson <
> robin.bartholdson@formulate.se> a écrit :
>
> > I submitted our patch in the mean time, do test it and see if it works
> for
> > you :-)
> >
> > https://github.com/apache/incubator-airflow/pull/2181
> > https://issues.apache.org/jira/browse/AIRFLOW-1034
> >
> > -Robin
> >
> > On 23 Mar 2017, at 17:43, Robin Bartholdson <
> > robin.bartholdson@formulate.se> wrote:
> >
> > There is some discussion in this JIRA,
> > https://issues.apache.org/jira/browse/AIRFLOW-115, about moving to boto3
> > and restructuring along the lines you discuss.
> >
> > I totally agree that it would indeed be very useful!
> >
> > -Robin
> >
> >
> > On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <
> > jalpedrinharamos@gmail.com> wrote:
> >
> > Hi Sumit,
> >
> > I was actually referring to what Robin (thank you) pointed out which is
> > reason enough for having region in S3Hook.
> >
> > There are even other situations having the region would be beneficial,
> > that's why the AwsHook has that definition, although I would agree that
> > their goals are different they typically refer to the same credentials,
> > just different use cases.
> >
> > This makes me feel that it becomes redundant to have both an S3
> connection
> > and an AWS connection with the same credentials just be able to access
> > region. Wouldn't it be beneficial to implement S3Hook (and S3 connection)
> > as a special case of AwsHook (and Aws connection) and be able to use both
> > interchangeably ?
> >
> > I'm also aware that one uses boto and the other uses boto3, and it seems
> > like the project is naturally slowly migrating towards the later. Maybe
> > there is even some discussion regarding this that I'm not aware.
> >
> > If you think this is a desirable move, I may start to design this move,
> > making sure nothing gets broken in the process.
> >
> > WDYT?
> >
> > On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson <
> > robin.bartholdson@formulate.se> wrote:
> >
> > I _think_ that’s a different issue (do correct me if I’m wrong)
> >
> > For a number of regions (see the list on
> > http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html),
> > you have to specify an extra host-parameter in the boto-S3Connection as
> > they only support signature V4 and the host-param is required to have
> this
> > working (
> >
> >
> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError
> > )
> >
> > -Robin
> >
> > On 23 Mar 2017, at 15:10, Sumit Maheshwari <su...@gmail.com>
> wrote:
> >
> > Are we talking about this
> > https://github.com/apache/incubator-airflow/pull/2176 ?
> >
> >
> >
> >
> > On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
> > robin.bartholdson@formulate.se> wrote:
> >
> > Hi,
> >
> > We actually have a patch for this that allows you to specify the host
> > parameter in the S3 config in the connections.
> >
> > Been thinking about submitting this for a while but have not had the
> time,
> > maybe now’s the time to do it :-)
> >
> > If you want to open a JIRA I can submit the patch we are using in
> > production!
> >
> > -Robin
> >
> >
> > On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
> > jalpedrinharamos@gmail.com> wrote:
> >
> > Hi,
> >
> > I've been using S3Hook for a while, but I've run into some issues for not
> > having a defined region. I can't really say why I can't access this
> bucket
> > without defining the region, but I know that there is no way to define
> > region in S3Hook via the connection extra parameter unless a config_file
> is
> > used, which is not always an available method.
> >
> > I haven't found any ticket regarding this, is there any alternative ?
> >
> > Regards,
> > Jorge
> >
> >
> >
> >
> >
> >
> >
> >
>
Re: Missing region in S3Hook when using extra
Posted by Gael Magnan <ga...@gmail.com>.
Hi,
we have had the same problem and basically did the same as Robin, the only
diff we have is we use the host from the connection instead of extra
params.
I can't agree more with the statement that we need to really rethink the
S3Hook, and maybe make some basic S3/AWS operators.
Gael
Le jeu. 23 mars 2017 à 18:05, Robin Bartholdson <
robin.bartholdson@formulate.se> a écrit :
> I submitted our patch in the mean time, do test it and see if it works for
> you :-)
>
> https://github.com/apache/incubator-airflow/pull/2181
> https://issues.apache.org/jira/browse/AIRFLOW-1034
>
> -Robin
>
> On 23 Mar 2017, at 17:43, Robin Bartholdson <
> robin.bartholdson@formulate.se> wrote:
>
> There is some discussion in this JIRA,
> https://issues.apache.org/jira/browse/AIRFLOW-115, about moving to boto3
> and restructuring along the lines you discuss.
>
> I totally agree that it would indeed be very useful!
>
> -Robin
>
>
> On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <
> jalpedrinharamos@gmail.com> wrote:
>
> Hi Sumit,
>
> I was actually referring to what Robin (thank you) pointed out which is
> reason enough for having region in S3Hook.
>
> There are even other situations having the region would be beneficial,
> that's why the AwsHook has that definition, although I would agree that
> their goals are different they typically refer to the same credentials,
> just different use cases.
>
> This makes me feel that it becomes redundant to have both an S3 connection
> and an AWS connection with the same credentials just be able to access
> region. Wouldn't it be beneficial to implement S3Hook (and S3 connection)
> as a special case of AwsHook (and Aws connection) and be able to use both
> interchangeably ?
>
> I'm also aware that one uses boto and the other uses boto3, and it seems
> like the project is naturally slowly migrating towards the later. Maybe
> there is even some discussion regarding this that I'm not aware.
>
> If you think this is a desirable move, I may start to design this move,
> making sure nothing gets broken in the process.
>
> WDYT?
>
> On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson <
> robin.bartholdson@formulate.se> wrote:
>
> I _think_ that’s a different issue (do correct me if I’m wrong)
>
> For a number of regions (see the list on
> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html),
> you have to specify an extra host-parameter in the boto-S3Connection as
> they only support signature V4 and the host-param is required to have this
> working (
>
> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError
> )
>
> -Robin
>
> On 23 Mar 2017, at 15:10, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Are we talking about this
> https://github.com/apache/incubator-airflow/pull/2176 ?
>
>
>
>
> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
> robin.bartholdson@formulate.se> wrote:
>
> Hi,
>
> We actually have a patch for this that allows you to specify the host
> parameter in the S3 config in the connections.
>
> Been thinking about submitting this for a while but have not had the time,
> maybe now’s the time to do it :-)
>
> If you want to open a JIRA I can submit the patch we are using in
> production!
>
> -Robin
>
>
> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
> jalpedrinharamos@gmail.com> wrote:
>
> Hi,
>
> I've been using S3Hook for a while, but I've run into some issues for not
> having a defined region. I can't really say why I can't access this bucket
> without defining the region, but I know that there is no way to define
> region in S3Hook via the connection extra parameter unless a config_file is
> used, which is not always an available method.
>
> I haven't found any ticket regarding this, is there any alternative ?
>
> Regards,
> Jorge
>
>
>
>
>
>
>
>
Re: Missing region in S3Hook when using extra
Posted by Robin Bartholdson <ro...@formulate.se>.
I submitted our patch in the mean time, do test it and see if it works for you :-)
https://github.com/apache/incubator-airflow/pull/2181
https://issues.apache.org/jira/browse/AIRFLOW-1034
-Robin
> On 23 Mar 2017, at 17:43, Robin Bartholdson <ro...@formulate.se> wrote:
>
> There is some discussion in this JIRA, https://issues.apache.org/jira/browse/AIRFLOW-115 <https://issues.apache.org/jira/browse/AIRFLOW-115>, about moving to boto3 and restructuring along the lines you discuss.
>
> I totally agree that it would indeed be very useful!
>
> -Robin
>
>
>> On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <jalpedrinharamos@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Sumit,
>>
>> I was actually referring to what Robin (thank you) pointed out which is
>> reason enough for having region in S3Hook.
>>
>> There are even other situations having the region would be beneficial,
>> that's why the AwsHook has that definition, although I would agree that
>> their goals are different they typically refer to the same credentials,
>> just different use cases.
>>
>> This makes me feel that it becomes redundant to have both an S3 connection
>> and an AWS connection with the same credentials just be able to access
>> region. Wouldn't it be beneficial to implement S3Hook (and S3 connection)
>> as a special case of AwsHook (and Aws connection) and be able to use both
>> interchangeably ?
>>
>> I'm also aware that one uses boto and the other uses boto3, and it seems
>> like the project is naturally slowly migrating towards the later. Maybe
>> there is even some discussion regarding this that I'm not aware.
>>
>> If you think this is a desirable move, I may start to design this move,
>> making sure nothing gets broken in the process.
>>
>> WDYT?
>>
>> On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson <
>> robin.bartholdson@formulate.se <ma...@formulate.se>> wrote:
>>
>>> I _think_ that’s a different issue (do correct me if I’m wrong)
>>>
>>> For a number of regions (see the list on
>>> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html <http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html>),
>>> you have to specify an extra host-parameter in the boto-S3Connection as
>>> they only support signature V4 and the host-param is required to have this
>>> working (
>>> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError <http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError>
>>> )
>>>
>>> -Robin
>>>
>>> On 23 Mar 2017, at 15:10, Sumit Maheshwari <su...@gmail.com> wrote:
>>>
>>> Are we talking about this
>>> https://github.com/apache/incubator-airflow/pull/2176 ?
>>>
>>>
>>>
>>>
>>> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
>>> robin.bartholdson@formulate.se> wrote:
>>>
>>> Hi,
>>>
>>> We actually have a patch for this that allows you to specify the host
>>> parameter in the S3 config in the connections.
>>>
>>> Been thinking about submitting this for a while but have not had the time,
>>> maybe now’s the time to do it :-)
>>>
>>> If you want to open a JIRA I can submit the patch we are using in
>>> production!
>>>
>>> -Robin
>>>
>>>
>>> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
>>> jalpedrinharamos@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I've been using S3Hook for a while, but I've run into some issues for not
>>> having a defined region. I can't really say why I can't access this bucket
>>> without defining the region, but I know that there is no way to define
>>> region in S3Hook via the connection extra parameter unless a config_file is
>>> used, which is not always an available method.
>>>
>>> I haven't found any ticket regarding this, is there any alternative ?
>>>
>>> Regards,
>>> Jorge
>>>
>>>
>>>
>>>
>>>
>
Re: Missing region in S3Hook when using extra
Posted by Robin Bartholdson <ro...@formulate.se>.
There is some discussion in this JIRA, https://issues.apache.org/jira/browse/AIRFLOW-115 <https://issues.apache.org/jira/browse/AIRFLOW-115>, about moving to boto3 and restructuring along the lines you discuss.
I totally agree that it would indeed be very useful!
-Robin
> On 23 Mar 2017, at 16:53, Jorge Alpedrinha Ramos <ja...@gmail.com> wrote:
>
> Hi Sumit,
>
> I was actually referring to what Robin (thank you) pointed out which is
> reason enough for having region in S3Hook.
>
> There are even other situations having the region would be beneficial,
> that's why the AwsHook has that definition, although I would agree that
> their goals are different they typically refer to the same credentials,
> just different use cases.
>
> This makes me feel that it becomes redundant to have both an S3 connection
> and an AWS connection with the same credentials just be able to access
> region. Wouldn't it be beneficial to implement S3Hook (and S3 connection)
> as a special case of AwsHook (and Aws connection) and be able to use both
> interchangeably ?
>
> I'm also aware that one uses boto and the other uses boto3, and it seems
> like the project is naturally slowly migrating towards the later. Maybe
> there is even some discussion regarding this that I'm not aware.
>
> If you think this is a desirable move, I may start to design this move,
> making sure nothing gets broken in the process.
>
> WDYT?
>
> On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson <
> robin.bartholdson@formulate.se> wrote:
>
>> I _think_ that’s a different issue (do correct me if I’m wrong)
>>
>> For a number of regions (see the list on
>> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html),
>> you have to specify an extra host-parameter in the boto-S3Connection as
>> they only support signature V4 and the host-param is required to have this
>> working (
>> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError
>> )
>>
>> -Robin
>>
>> On 23 Mar 2017, at 15:10, Sumit Maheshwari <su...@gmail.com> wrote:
>>
>> Are we talking about this
>> https://github.com/apache/incubator-airflow/pull/2176 ?
>>
>>
>>
>>
>> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
>> robin.bartholdson@formulate.se> wrote:
>>
>> Hi,
>>
>> We actually have a patch for this that allows you to specify the host
>> parameter in the S3 config in the connections.
>>
>> Been thinking about submitting this for a while but have not had the time,
>> maybe now’s the time to do it :-)
>>
>> If you want to open a JIRA I can submit the patch we are using in
>> production!
>>
>> -Robin
>>
>>
>> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
>> jalpedrinharamos@gmail.com> wrote:
>>
>> Hi,
>>
>> I've been using S3Hook for a while, but I've run into some issues for not
>> having a defined region. I can't really say why I can't access this bucket
>> without defining the region, but I know that there is no way to define
>> region in S3Hook via the connection extra parameter unless a config_file is
>> used, which is not always an available method.
>>
>> I haven't found any ticket regarding this, is there any alternative ?
>>
>> Regards,
>> Jorge
>>
>>
>>
>>
>>
Re: Missing region in S3Hook when using extra
Posted by Jorge Alpedrinha Ramos <ja...@gmail.com>.
Hi Sumit,
I was actually referring to what Robin (thank you) pointed out which is
reason enough for having region in S3Hook.
There are even other situations having the region would be beneficial,
that's why the AwsHook has that definition, although I would agree that
their goals are different they typically refer to the same credentials,
just different use cases.
This makes me feel that it becomes redundant to have both an S3 connection
and an AWS connection with the same credentials just be able to access
region. Wouldn't it be beneficial to implement S3Hook (and S3 connection)
as a special case of AwsHook (and Aws connection) and be able to use both
interchangeably ?
I'm also aware that one uses boto and the other uses boto3, and it seems
like the project is naturally slowly migrating towards the later. Maybe
there is even some discussion regarding this that I'm not aware.
If you think this is a desirable move, I may start to design this move,
making sure nothing gets broken in the process.
WDYT?
On Thu, Mar 23, 2017 at 2:46 PM Robin Bartholdson <
robin.bartholdson@formulate.se> wrote:
> I _think_ that’s a different issue (do correct me if I’m wrong)
>
> For a number of regions (see the list on
> http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html),
> you have to specify an extra host-parameter in the boto-S3Connection as
> they only support signature V4 and the host-param is required to have this
> working (
> http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError
> )
>
> -Robin
>
> On 23 Mar 2017, at 15:10, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Are we talking about this
> https://github.com/apache/incubator-airflow/pull/2176 ?
>
>
>
>
> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
> robin.bartholdson@formulate.se> wrote:
>
> Hi,
>
> We actually have a patch for this that allows you to specify the host
> parameter in the S3 config in the connections.
>
> Been thinking about submitting this for a while but have not had the time,
> maybe now’s the time to do it :-)
>
> If you want to open a JIRA I can submit the patch we are using in
> production!
>
> -Robin
>
>
> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
> jalpedrinharamos@gmail.com> wrote:
>
> Hi,
>
> I've been using S3Hook for a while, but I've run into some issues for not
> having a defined region. I can't really say why I can't access this bucket
> without defining the region, but I know that there is no way to define
> region in S3Hook via the connection extra parameter unless a config_file is
> used, which is not always an available method.
>
> I haven't found any ticket regarding this, is there any alternative ?
>
> Regards,
> Jorge
>
>
>
>
>
Re: Missing region in S3Hook when using extra
Posted by Robin Bartholdson <ro...@formulate.se>.
I _think_ that’s a different issue (do correct me if I’m wrong)
For a number of regions (see the list on http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html <http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html>), you have to specify an extra host-parameter in the boto-S3Connection as they only support signature V4 and the host-param is required to have this working (http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError <http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.connection.HostRequiredError>)
-Robin
> On 23 Mar 2017, at 15:10, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Are we talking about this
> https://github.com/apache/incubator-airflow/pull/2176 ?
>
>
>
>
> On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
> robin.bartholdson@formulate.se> wrote:
>
>> Hi,
>>
>> We actually have a patch for this that allows you to specify the host
>> parameter in the S3 config in the connections.
>>
>> Been thinking about submitting this for a while but have not had the time,
>> maybe now’s the time to do it :-)
>>
>> If you want to open a JIRA I can submit the patch we are using in
>> production!
>>
>> -Robin
>>
>>
>> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
>> jalpedrinharamos@gmail.com> wrote:
>>
>> Hi,
>>
>> I've been using S3Hook for a while, but I've run into some issues for not
>> having a defined region. I can't really say why I can't access this bucket
>> without defining the region, but I know that there is no way to define
>> region in S3Hook via the connection extra parameter unless a config_file is
>> used, which is not always an available method.
>>
>> I haven't found any ticket regarding this, is there any alternative ?
>>
>> Regards,
>> Jorge
>>
>>
>>
Re: Missing region in S3Hook when using extra
Posted by Sumit Maheshwari <su...@gmail.com>.
Are we talking about this
https://github.com/apache/incubator-airflow/pull/2176 ?
On Thu, Mar 23, 2017 at 4:32 PM, Robin Bartholdson <
robin.bartholdson@formulate.se> wrote:
> Hi,
>
> We actually have a patch for this that allows you to specify the host
> parameter in the S3 config in the connections.
>
> Been thinking about submitting this for a while but have not had the time,
> maybe now’s the time to do it :-)
>
> If you want to open a JIRA I can submit the patch we are using in
> production!
>
> -Robin
>
>
> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <
> jalpedrinharamos@gmail.com> wrote:
>
> Hi,
>
> I've been using S3Hook for a while, but I've run into some issues for not
> having a defined region. I can't really say why I can't access this bucket
> without defining the region, but I know that there is no way to define
> region in S3Hook via the connection extra parameter unless a config_file is
> used, which is not always an available method.
>
> I haven't found any ticket regarding this, is there any alternative ?
>
> Regards,
> Jorge
>
>
>
Re: Missing region in S3Hook when using extra
Posted by Robin Bartholdson <ro...@formulate.se>.
Hi,
We actually have a patch for this that allows you to specify the host parameter in the S3 config in the connections.
Been thinking about submitting this for a while but have not had the time, maybe now’s the time to do it :-)
If you want to open a JIRA I can submit the patch we are using in production!
-Robin
> On 23 Mar 2017, at 11:55, Jorge Alpedrinha Ramos <ja...@gmail.com> wrote:
>
> Hi,
>
> I've been using S3Hook for a while, but I've run into some issues for not
> having a defined region. I can't really say why I can't access this bucket
> without defining the region, but I know that there is no way to define
> region in S3Hook via the connection extra parameter unless a config_file is
> used, which is not always an available method.
>
> I haven't found any ticket regarding this, is there any alternative ?
>
> Regards,
> Jorge