You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by John Reynolds <jr...@vpicu.net> on 2015/03/10 23:48:24 UTC

Amazon S3 data transfer for filemanager

Hi,
i’m new to the terminology of OODT (and OODT) so please bear with me.
i’m looking to do a couple things with OODT and integration with AWS, and wanted to see if anyone has done this already.
(I searched through the mail archive, and the wiki, and havent found quite what i’m looking for, other than some references to an S3 Datatransferer, that may or may not be complete)

So, what i’m looking to do:
1. ingest files into oodt file manager with S3 as the source
2. store files in S3 that have been ingested by file manager (from an arbitrary source)

what would be the general strategy for doing each of those things (what classes need to be implemented / extended)
or if someone has done this already, let me know

Thanks!
John Reynolds



Re: Amazon S3 data transfer for filemanager

Posted by John Reynolds <jr...@vpicu.net>.
Thanks Michael!
i didnt swap out openjdk for oracle version, i’ll try that next.

> On Mar 13, 2015, at 9:10 AM, Michael Starch <st...@umich.edu> wrote:
> 
> John,
> 
> Did you remove the code at the top of the script that sets JAVA_EXT_JARS?
> Are you running the oracle java still?
> 
> If both are true, can you send me the latest error?
> 
> -Michael
> 
> 
> 
> On Thu, Mar 12, 2015 at 5:14 PM, John Reynolds <jr...@vpicu.net> wrote:
> 
>> Thanks Michael, this works for me (filemanager starts), however i’m still
>> getting the same datatransferer error
>> 
>>> On Mar 12, 2015, at 1:52 PM, Michael Starch <st...@umich.edu> wrote:
>>> 
>>> John,
>>> 
>>> I should be more verbose.  The java classpath traditionally did not pick
>> up
>>> multiple jars, so it was super labor intensive to setup.  These days you
>>> can use * inside the classpath to pick up multiple jars....but it must be
>>> in " " because otherwise the shell will glob the * if it is outside of
>>> quotes.
>>> 
>>> If this doesn't work, try the other recommendations in:
>>> 
>>> 
>>> 
>> http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath
>>> 
>>> -Michael
>>> 
>>> 
>>> On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <st...@umich.edu>
>> wrote:
>>> 
>>>> John,
>>>> 
>>>> Change:
>>>>   -classpath "$FILEMGR_HOME"/lib \
>>>> To:
>>>>   -classpath "$FILEMGR_HOME/lib/*" \
>>>> 
>>>> -Michael
>>>> 
>>>> 
>>>> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <jr...@vpicu.net>
>>>> wrote:
>>>> 
>>>>> Thanks Michael, if i modify the filemgr-client to look like this (at
>> the
>>>>> end)
>>>>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
>>>>> -classpath "$FILEMGR_HOME"/lib \
>>>>> 
>>>>> 
>> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties
>>>>> \
>>>>> 
>> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \
>>>>> 
>>>>> 
>> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml
>>>>> \
>>>>> 
>>>>> 
>> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml
>>>>> \
>>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“
>>>>> 
>>>>> (replacing ext jars with -classpath) then i get
>>>>> Error: Could not find or load main class
>>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
>>>>> i assume i’m doing something wrong with classpath but not sure what
>>>>> 
>>>>>> On Mar 12, 2015, at 11:31 AM, Michael Starch <st...@umich.edu>
>>>>> wrote:
>>>>>> 
>>>>>> John,
>>>>>> 
>>>>>> Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
>>>>>> there.  If so, it is clobbering the default path for "extension" jars,
>>>>> and
>>>>>> your java encryption jars are not being picked up. If it does set
>>>>>> JAVA_EXT_JARS you have two options:
>>>>>> 
>>>>>> 1. Move all your encryption jars into FILEMGR_HOME/lib/
>>>>>> 2. update filemgr-client script to us classpath to specify the jars in
>>>>> the
>>>>>> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS
>>>>>> 
>>>>>> 
>>>>>> -Michael
>>>>>> 
>>>>>> 
>>>>>> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jr...@vpicu.net>
>>>>> wrote:
>>>>>> 
>>>>>>> Hi Michael
>>>>>>> yeah it’s openjdk 1.7 (“1.7.0_75")
>>>>>>> i did download the the unlimited encryption jar from oracle and
>>>>> replaced
>>>>>>> the local_policy / us_export_policy jars in javahome/jre/lib/security
>>>>>>> more i read, maybe limited by jce.jar
>>>>>>> 
>>>>>>> i dont have anything special set for extension jars
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu>
>>>>> wrote:
>>>>>>>> 
>>>>>>>> John,
>>>>>>>> 
>>>>>>>> What version of the JDK are you running, and what is your extension
>>>>> jars
>>>>>>>> environment variable set to.  Do you have the java cryptology jar
>>>>>>> included
>>>>>>>> (Oracle JDK usually has this, I don't know if Open JDK does).
>>>>>>>> 
>>>>>>>> "Algorithm HmacSHA1 not available" is usually thrown when Java
>> cannot
>>>>>>> find
>>>>>>>> the java crypto jar used to calculate the given hash.
>>>>>>>> 
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jreynolds@vpicu.net
>>> 
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Lewis,
>>>>>>>>> using the latest docker buggtb/oodt image, which i assume is .8
>>>>>>>>> here’s the command i’m running to test the upload
>>>>>>>>> 
>>>>>>>>> filemgr-client --url http://localhost:9000 --operation
>>>>> --ingestProduct
>>>>>>>>> --productName test --productStructure Flat --productTypeName
>>>>> GenericFile
>>>>>>>>> --metadataFile file:///root/test.txt.met --refs
>> file:///root/test.txt
>>>>>>>>> 
>>>>>>>>> i verified that i can upload to the path using the s3 tools on the
>>>>> box /
>>>>>>>>> with same credentials i put in the properties file
>>>>>>>>> 
>>>>>>>>> here’s the full exception returned:
>>>>>>>>> 
>>>>>>>>> 
>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>>>> 
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
>>>>>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>     at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>     at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>>>>>>>>>     at
>>>>>>>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>>>>>>>>>     at
>>>>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>>>>>>>>>     at
>>>>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>>>>>>>>>     at
>>>>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>>>>>>>>>     at
>>>>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>>>>>>>>>     at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>>>>>>>>>     at java.lang.Thread.run(Thread.java:745)
>>>>>>>>> Caused by:
>>>>>>>>> 
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
>>>>>>>>>     ... 12 more
>>>>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to
>> calculate a
>>>>>>>>> request signature: Unable to calculate a request signature:
>> Algorithm
>>>>>>>>> HmacSHA1 not available
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
>>>>>>>>>     at
>>>>>>>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
>>>>>>>>>     at
>>>>>>>>> 
>>>>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
>>>>>>>>>     ... 13 more
>>>>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to
>> calculate a
>>>>>>>>> request signature: Algorithm HmacSHA1 not available
>>>>>>>>>     at
>>>>>>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
>>>>>>>>>     ... 20 more
>>>>>>>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm
>> HmacSHA1
>>>>>>> not
>>>>>>>>> available
>>>>>>>>>     at javax.crypto.Mac.getInstance(Mac.java:176)
>>>>>>>>>     at
>>>>>>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
>>>>>>>>>     ... 21 more
>>>>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>> Error
>>>>>>>>> ingesting product
>>>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>>>>>>> ]
>>>>>>>>> :
>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>>>>>>>     at
>>>>>>>>> 
>>>>> 
>> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>>>>>>>     at
>>>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>>>>>     at
>>>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>>>>>     at
>>>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>>>>>>>>     at
>>>>>>>>> 
>>>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>>>>>     at
>>>>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>>>>>>>>> java.lang.Exception: Failed to ingest product
>>>>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>>>>>>>>> java.lang.Exception:
>>>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>> Error
>>>>>>>>> ingesting product
>>>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>>>>>>> ]
>>>>>>>>> :
>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>>>>>>>>     at
>>>>>>>>> 
>>>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>>>>>     at
>>>>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>>>>>     at
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>>>>>>>>> ERROR: Failed to ingest product 'test' : Failed to ingest product
>>>>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>>>>>>>>> java.lang.Exception:
>>>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>> Error
>>>>>>>>> ingesting product
>>>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>>>>>>> ]
>>>>>>>>> :
>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
>>>>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi John,
>>>>>>>>>> What beau on of OODT are you using?
>>>>>>>>>> The file types should be encapsulated by the factory
>> implementation
>>>>> I
>>>>>>>>> would
>>>>>>>>>> guess. It should be. Case of mike this the default filemgr store
>>>>> then
>>>>>>> off
>>>>>>>>>> you go.
>>>>>>>>>> Can you provide a paste of how your arriving at your exception
>>>>> please?
>>>>>>>>>> Lewia
>>>>>>>>>> 
>>>>>>>>>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Should S3DataTransferer work with the generic file type from the
>>>>> get
>>>>>>> go
>>>>>>>>> or
>>>>>>>>>>> is there anything i need to change
>>>>>>>>>>> i was using the radix docker but getting 'Caused by:
>>>>>>>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
>>>>>>>>> available’
>>>>>>>>>>> i tried downloading the jce but no dice
>>>>>>>>>>> i might try just building on my local machine and trying again
>>>>>>>>>>> 
>>>>>>>>>>> Thanks
>>>>>>>>>>> 
>>>>>>>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>>>>>>>>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey John,
>>>>>>>>>>>> You're right, you've nearly answered everything here ;)
>>>>>>>>>>>> OK, so your properties can be located here
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>>>>>>>>>>>> You'll see the usual key, value pairs in there which you can ad
>>>>>>> before
>>>>>>>>>>>> build the project and packaging filemgr.properties along with
>> your
>>>>>>>>>>> compiled
>>>>>>>>>>>> code.
>>>>>>>>>>>> Regarding the following,
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <
>>>>> jreynolds@vpicu.net
>>>>>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> in the near future, i will want to pass in a security token as
>>>>> well
>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer
>>>>>>> class
>>>>>>>>>>>>> (which expires after an hour), what’s the best practice to read
>>>>> in
>>>>>>>>> this
>>>>>>>>>>>>> transient data from the command line (or wherever else i would
>>>>>>>>> initiate
>>>>>>>>>>> a
>>>>>>>>>>>>> job from) since this wouldnt go in a properties file
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Do you mean that your security token expires hourly? As oppose
>> to
>>>>>>> your
>>>>>>>>> s3
>>>>>>>>>>>> client?
>>>>>>>>>>>> Lets try and defined exactly what you are after here.
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Lewis
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> *Lewis*
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 


Re: Amazon S3 data transfer for filemanager

Posted by Michael Starch <st...@umich.edu>.
John,

Did you remove the code at the top of the script that sets JAVA_EXT_JARS?
Are you running the oracle java still?

If both are true, can you send me the latest error?

-Michael



On Thu, Mar 12, 2015 at 5:14 PM, John Reynolds <jr...@vpicu.net> wrote:

> Thanks Michael, this works for me (filemanager starts), however i’m still
> getting the same datatransferer error
>
> > On Mar 12, 2015, at 1:52 PM, Michael Starch <st...@umich.edu> wrote:
> >
> > John,
> >
> > I should be more verbose.  The java classpath traditionally did not pick
> up
> > multiple jars, so it was super labor intensive to setup.  These days you
> > can use * inside the classpath to pick up multiple jars....but it must be
> > in " " because otherwise the shell will glob the * if it is outside of
> > quotes.
> >
> > If this doesn't work, try the other recommendations in:
> >
> >
> >
> http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath
> >
> > -Michael
> >
> >
> > On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <st...@umich.edu>
> wrote:
> >
> >> John,
> >>
> >> Change:
> >>    -classpath "$FILEMGR_HOME"/lib \
> >> To:
> >>    -classpath "$FILEMGR_HOME/lib/*" \
> >>
> >> -Michael
> >>
> >>
> >> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <jr...@vpicu.net>
> >> wrote:
> >>
> >>> Thanks Michael, if i modify the filemgr-client to look like this (at
> the
> >>> end)
> >>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
> >>>  -classpath "$FILEMGR_HOME"/lib \
> >>>
> >>>
> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties
> >>> \
> >>>
> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \
> >>>
> >>>
> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml
> >>> \
> >>>
> >>>
> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml
> >>> \
> >>>  org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“
> >>>
> >>> (replacing ext jars with -classpath) then i get
> >>> Error: Could not find or load main class
> >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
> >>> i assume i’m doing something wrong with classpath but not sure what
> >>>
> >>>> On Mar 12, 2015, at 11:31 AM, Michael Starch <st...@umich.edu>
> >>> wrote:
> >>>>
> >>>> John,
> >>>>
> >>>> Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
> >>>> there.  If so, it is clobbering the default path for "extension" jars,
> >>> and
> >>>> your java encryption jars are not being picked up. If it does set
> >>>> JAVA_EXT_JARS you have two options:
> >>>>
> >>>> 1. Move all your encryption jars into FILEMGR_HOME/lib/
> >>>> 2. update filemgr-client script to us classpath to specify the jars in
> >>> the
> >>>> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS
> >>>>
> >>>>
> >>>> -Michael
> >>>>
> >>>>
> >>>> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jr...@vpicu.net>
> >>> wrote:
> >>>>
> >>>>> Hi Michael
> >>>>> yeah it’s openjdk 1.7 (“1.7.0_75")
> >>>>> i did download the the unlimited encryption jar from oracle and
> >>> replaced
> >>>>> the local_policy / us_export_policy jars in javahome/jre/lib/security
> >>>>> more i read, maybe limited by jce.jar
> >>>>>
> >>>>> i dont have anything special set for extension jars
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu>
> >>> wrote:
> >>>>>>
> >>>>>> John,
> >>>>>>
> >>>>>> What version of the JDK are you running, and what is your extension
> >>> jars
> >>>>>> environment variable set to.  Do you have the java cryptology jar
> >>>>> included
> >>>>>> (Oracle JDK usually has this, I don't know if Open JDK does).
> >>>>>>
> >>>>>> "Algorithm HmacSHA1 not available" is usually thrown when Java
> cannot
> >>>>> find
> >>>>>> the java crypto jar used to calculate the given hash.
> >>>>>>
> >>>>>> -Michael
> >>>>>>
> >>>>>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jreynolds@vpicu.net
> >
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hi Lewis,
> >>>>>>> using the latest docker buggtb/oodt image, which i assume is .8
> >>>>>>> here’s the command i’m running to test the upload
> >>>>>>>
> >>>>>>> filemgr-client --url http://localhost:9000 --operation
> >>> --ingestProduct
> >>>>>>> --productName test --productStructure Flat --productTypeName
> >>> GenericFile
> >>>>>>> --metadataFile file:///root/test.txt.met --refs
> file:///root/test.txt
> >>>>>>>
> >>>>>>> i verified that i can upload to the path using the s3 tools on the
> >>> box /
> >>>>>>> with same credentials i put in the properties file
> >>>>>>>
> >>>>>>> here’s the full exception returned:
> >>>>>>>
> >>>>>>>
> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>>>>>
> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>>>>> Failed to upload product reference /root/test.txt to S3 at
> >>>>>>> usr/src/oodt/data/archive/test/test.txt
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
> >>>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>>>>      at java.lang.reflect.Method.invoke(Method.java:606)
> >>>>>>>      at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >>>>>>>      at
> >>>>>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >>>>>>>      at
> >>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >>>>>>>      at
> >>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >>>>>>>      at
> >>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >>>>>>>      at
> >>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >>>>>>>      at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >>>>>>>      at java.lang.Thread.run(Thread.java:745)
> >>>>>>> Caused by:
> >>>>>>>
> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>>>>> Failed to upload product reference /root/test.txt to S3 at
> >>>>>>> usr/src/oodt/data/archive/test/test.txt
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
> >>>>>>>      ... 12 more
> >>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to
> calculate a
> >>>>>>> request signature: Unable to calculate a request signature:
> Algorithm
> >>>>>>> HmacSHA1 not available
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
> >>>>>>>      at
> >>>>>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
> >>>>>>>      at
> >>>>>>>
> >>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
> >>>>>>>      ... 13 more
> >>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to
> calculate a
> >>>>>>> request signature: Algorithm HmacSHA1 not available
> >>>>>>>      at
> >>>>>>>
> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
> >>>>>>>      ... 20 more
> >>>>>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm
> HmacSHA1
> >>>>> not
> >>>>>>> available
> >>>>>>>      at javax.crypto.Mac.getInstance(Mac.java:176)
> >>>>>>>      at
> >>>>>>>
> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
> >>>>>>>      ... 21 more
> >>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
> >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >>> Error
> >>>>>>> ingesting product
> >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> >>>>> ]
> >>>>>>> :
> >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>>>>> Failed to upload product reference /root/test.txt to S3 at
> >>>>>>> usr/src/oodt/data/archive/test/test.txt
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
> >>>>>>>      at
> >>>>>>>
> >>>
> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> >>>>>>>      at
> >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> >>>>>>>      at
> >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> >>>>>>>      at
> >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
> >>>>>>>      at
> >>>>>>>
> >>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> >>>>>>>      at
> >>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> >>>>>>> java.lang.Exception: Failed to ingest product
> >>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> >>>>>>> java.lang.Exception:
> >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >>> Error
> >>>>>>> ingesting product
> >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> >>>>> ]
> >>>>>>> :
> >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>>>>> Failed to upload product reference /root/test.txt to S3 at
> >>>>>>> usr/src/oodt/data/archive/test/test.txt
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
> >>>>>>>      at
> >>>>>>>
> >>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> >>>>>>>      at
> >>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> >>>>>>>      at
> >>>>>>>
> >>>>>
> >>>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> >>>>>>> ERROR: Failed to ingest product 'test' : Failed to ingest product
> >>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> >>>>>>> java.lang.Exception:
> >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >>> Error
> >>>>>>> ingesting product
> >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> >>>>> ]
> >>>>>>> :
> >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>>>>> Failed to upload product reference /root/test.txt to S3 at
> >>>>>>> usr/src/oodt/data/archive/test/test.txt
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
> >>>>>>> lewis.mcgibbney@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi John,
> >>>>>>>> What beau on of OODT are you using?
> >>>>>>>> The file types should be encapsulated by the factory
> implementation
> >>> I
> >>>>>>> would
> >>>>>>>> guess. It should be. Case of mike this the default filemgr store
> >>> then
> >>>>> off
> >>>>>>>> you go.
> >>>>>>>> Can you provide a paste of how your arriving at your exception
> >>> please?
> >>>>>>>> Lewia
> >>>>>>>>
> >>>>>>>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Should S3DataTransferer work with the generic file type from the
> >>> get
> >>>>> go
> >>>>>>> or
> >>>>>>>>> is there anything i need to change
> >>>>>>>>> i was using the radix docker but getting 'Caused by:
> >>>>>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
> >>>>>>> available’
> >>>>>>>>> i tried downloading the jce but no dice
> >>>>>>>>> i might try just building on my local machine and trying again
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
> >>>>>>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hey John,
> >>>>>>>>>> You're right, you've nearly answered everything here ;)
> >>>>>>>>>> OK, so your properties can be located here
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
> >>>>>>>>>> You'll see the usual key, value pairs in there which you can ad
> >>>>> before
> >>>>>>>>>> build the project and packaging filemgr.properties along with
> your
> >>>>>>>>> compiled
> >>>>>>>>>> code.
> >>>>>>>>>> Regarding the following,
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <
> >>> jreynolds@vpicu.net
> >>>>>>>>> <javascript:;>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> in the near future, i will want to pass in a security token as
> >>> well
> >>>>> to
> >>>>>>>>> the
> >>>>>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer
> >>>>> class
> >>>>>>>>>>> (which expires after an hour), what’s the best practice to read
> >>> in
> >>>>>>> this
> >>>>>>>>>>> transient data from the command line (or wherever else i would
> >>>>>>> initiate
> >>>>>>>>> a
> >>>>>>>>>>> job from) since this wouldnt go in a properties file
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Do you mean that your security token expires hourly? As oppose
> to
> >>>>> your
> >>>>>>> s3
> >>>>>>>>>> client?
> >>>>>>>>>> Lets try and defined exactly what you are after here.
> >>>>>>>>>> Thanks
> >>>>>>>>>> Lewis
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> *Lewis*
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: Amazon S3 data transfer for filemanager

Posted by John Reynolds <jr...@vpicu.net>.
Thanks Michael, this works for me (filemanager starts), however i’m still getting the same datatransferer error

> On Mar 12, 2015, at 1:52 PM, Michael Starch <st...@umich.edu> wrote:
> 
> John,
> 
> I should be more verbose.  The java classpath traditionally did not pick up
> multiple jars, so it was super labor intensive to setup.  These days you
> can use * inside the classpath to pick up multiple jars....but it must be
> in " " because otherwise the shell will glob the * if it is outside of
> quotes.
> 
> If this doesn't work, try the other recommendations in:
> 
> 
> http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath
> 
> -Michael
> 
> 
> On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <st...@umich.edu> wrote:
> 
>> John,
>> 
>> Change:
>>    -classpath "$FILEMGR_HOME"/lib \
>> To:
>>    -classpath "$FILEMGR_HOME/lib/*" \
>> 
>> -Michael
>> 
>> 
>> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <jr...@vpicu.net>
>> wrote:
>> 
>>> Thanks Michael, if i modify the filemgr-client to look like this (at the
>>> end)
>>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
>>>  -classpath "$FILEMGR_HOME"/lib \
>>> 
>>> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties
>>> \
>>>  -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \
>>> 
>>> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml
>>> \
>>> 
>>> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml
>>> \
>>>  org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“
>>> 
>>> (replacing ext jars with -classpath) then i get
>>> Error: Could not find or load main class
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
>>> i assume i’m doing something wrong with classpath but not sure what
>>> 
>>>> On Mar 12, 2015, at 11:31 AM, Michael Starch <st...@umich.edu>
>>> wrote:
>>>> 
>>>> John,
>>>> 
>>>> Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
>>>> there.  If so, it is clobbering the default path for "extension" jars,
>>> and
>>>> your java encryption jars are not being picked up. If it does set
>>>> JAVA_EXT_JARS you have two options:
>>>> 
>>>> 1. Move all your encryption jars into FILEMGR_HOME/lib/
>>>> 2. update filemgr-client script to us classpath to specify the jars in
>>> the
>>>> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS
>>>> 
>>>> 
>>>> -Michael
>>>> 
>>>> 
>>>> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jr...@vpicu.net>
>>> wrote:
>>>> 
>>>>> Hi Michael
>>>>> yeah it’s openjdk 1.7 (“1.7.0_75")
>>>>> i did download the the unlimited encryption jar from oracle and
>>> replaced
>>>>> the local_policy / us_export_policy jars in javahome/jre/lib/security
>>>>> more i read, maybe limited by jce.jar
>>>>> 
>>>>> i dont have anything special set for extension jars
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu>
>>> wrote:
>>>>>> 
>>>>>> John,
>>>>>> 
>>>>>> What version of the JDK are you running, and what is your extension
>>> jars
>>>>>> environment variable set to.  Do you have the java cryptology jar
>>>>> included
>>>>>> (Oracle JDK usually has this, I don't know if Open JDK does).
>>>>>> 
>>>>>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot
>>>>> find
>>>>>> the java crypto jar used to calculate the given hash.
>>>>>> 
>>>>>> -Michael
>>>>>> 
>>>>>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jr...@vpicu.net>
>>>>> wrote:
>>>>>> 
>>>>>>> Hi Lewis,
>>>>>>> using the latest docker buggtb/oodt image, which i assume is .8
>>>>>>> here’s the command i’m running to test the upload
>>>>>>> 
>>>>>>> filemgr-client --url http://localhost:9000 --operation
>>> --ingestProduct
>>>>>>> --productName test --productStructure Flat --productTypeName
>>> GenericFile
>>>>>>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
>>>>>>> 
>>>>>>> i verified that i can upload to the path using the s3 tools on the
>>> box /
>>>>>>> with same credentials i put in the properties file
>>>>>>> 
>>>>>>> here’s the full exception returned:
>>>>>>> 
>>>>>>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
>>>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>      at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>      at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>>>>>>>      at
>>>>>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>>>>>>>      at
>>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>>>>>>>      at
>>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>>>>>>>      at
>>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>>>>>>>      at
>>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>>>>>>>      at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>>>>>>>      at java.lang.Thread.run(Thread.java:745)
>>>>>>> Caused by:
>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
>>>>>>>      ... 12 more
>>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>>>>>>> request signature: Unable to calculate a request signature: Algorithm
>>>>>>> HmacSHA1 not available
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
>>>>>>>      at
>>>>>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
>>>>>>>      at
>>>>>>> 
>>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
>>>>>>>      ... 13 more
>>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>>>>>>> request signature: Algorithm HmacSHA1 not available
>>>>>>>      at
>>>>>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
>>>>>>>      ... 20 more
>>>>>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1
>>>>> not
>>>>>>> available
>>>>>>>      at javax.crypto.Mac.getInstance(Mac.java:176)
>>>>>>>      at
>>>>>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
>>>>>>>      ... 21 more
>>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>> Error
>>>>>>> ingesting product
>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>>>>> ]
>>>>>>> :
>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>>>>>      at
>>>>>>> 
>>> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>>>>>      at
>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>>>      at
>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>>>      at
>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>>>>>>      at
>>>>>>> 
>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>>>      at
>>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>>>>>>> java.lang.Exception: Failed to ingest product
>>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>>>>>>> java.lang.Exception:
>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>> Error
>>>>>>> ingesting product
>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>>>>> ]
>>>>>>> :
>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>>>>>>      at
>>>>>>> 
>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>>>      at
>>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>>>      at
>>>>>>> 
>>>>> 
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>>>>>>> ERROR: Failed to ingest product 'test' : Failed to ingest product
>>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>>>>>>> java.lang.Exception:
>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>> Error
>>>>>>> ingesting product
>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>>>>> ]
>>>>>>> :
>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>>>>> Failed to upload product reference /root/test.txt to S3 at
>>>>>>> usr/src/oodt/data/archive/test/test.txt
>>>>>>> 
>>>>>>> 
>>>>>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
>>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Hi John,
>>>>>>>> What beau on of OODT are you using?
>>>>>>>> The file types should be encapsulated by the factory implementation
>>> I
>>>>>>> would
>>>>>>>> guess. It should be. Case of mike this the default filemgr store
>>> then
>>>>> off
>>>>>>>> you go.
>>>>>>>> Can you provide a paste of how your arriving at your exception
>>> please?
>>>>>>>> Lewia
>>>>>>>> 
>>>>>>>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Should S3DataTransferer work with the generic file type from the
>>> get
>>>>> go
>>>>>>> or
>>>>>>>>> is there anything i need to change
>>>>>>>>> i was using the radix docker but getting 'Caused by:
>>>>>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
>>>>>>> available’
>>>>>>>>> i tried downloading the jce but no dice
>>>>>>>>> i might try just building on my local machine and trying again
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> 
>>>>>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>>>>>>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hey John,
>>>>>>>>>> You're right, you've nearly answered everything here ;)
>>>>>>>>>> OK, so your properties can be located here
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>>>>>>>>>> You'll see the usual key, value pairs in there which you can ad
>>>>> before
>>>>>>>>>> build the project and packaging filemgr.properties along with your
>>>>>>>>> compiled
>>>>>>>>>> code.
>>>>>>>>>> Regarding the following,
>>>>>>>>>> 
>>>>>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <
>>> jreynolds@vpicu.net
>>>>>>>>> <javascript:;>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> in the near future, i will want to pass in a security token as
>>> well
>>>>> to
>>>>>>>>> the
>>>>>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer
>>>>> class
>>>>>>>>>>> (which expires after an hour), what’s the best practice to read
>>> in
>>>>>>> this
>>>>>>>>>>> transient data from the command line (or wherever else i would
>>>>>>> initiate
>>>>>>>>> a
>>>>>>>>>>> job from) since this wouldnt go in a properties file
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Do you mean that your security token expires hourly? As oppose to
>>>>> your
>>>>>>> s3
>>>>>>>>>> client?
>>>>>>>>>> Lets try and defined exactly what you are after here.
>>>>>>>>>> Thanks
>>>>>>>>>> Lewis
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> *Lewis*
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>> 


Re: Amazon S3 data transfer for filemanager

Posted by Michael Starch <st...@umich.edu>.
John,

I should be more verbose.  The java classpath traditionally did not pick up
multiple jars, so it was super labor intensive to setup.  These days you
can use * inside the classpath to pick up multiple jars....but it must be
in " " because otherwise the shell will glob the * if it is outside of
quotes.

If this doesn't work, try the other recommendations in:


http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath

-Michael


On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <st...@umich.edu> wrote:

> John,
>
> Change:
>     -classpath "$FILEMGR_HOME"/lib \
> To:
>     -classpath "$FILEMGR_HOME/lib/*" \
>
> -Michael
>
>
> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <jr...@vpicu.net>
> wrote:
>
>> Thanks Michael, if i modify the filemgr-client to look like this (at the
>> end)
>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
>>   -classpath "$FILEMGR_HOME"/lib \
>>
>> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties
>> \
>>   -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \
>>
>> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml
>> \
>>
>> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml
>> \
>>   org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“
>>
>> (replacing ext jars with -classpath) then i get
>> Error: Could not find or load main class
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
>> i assume i’m doing something wrong with classpath but not sure what
>>
>> > On Mar 12, 2015, at 11:31 AM, Michael Starch <st...@umich.edu>
>> wrote:
>> >
>> > John,
>> >
>> > Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
>> > there.  If so, it is clobbering the default path for "extension" jars,
>> and
>> > your java encryption jars are not being picked up. If it does set
>> > JAVA_EXT_JARS you have two options:
>> >
>> > 1. Move all your encryption jars into FILEMGR_HOME/lib/
>> > 2. update filemgr-client script to us classpath to specify the jars in
>> the
>> > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS
>> >
>> >
>> > -Michael
>> >
>> >
>> > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jr...@vpicu.net>
>> wrote:
>> >
>> >> Hi Michael
>> >> yeah it’s openjdk 1.7 (“1.7.0_75")
>> >> i did download the the unlimited encryption jar from oracle and
>> replaced
>> >> the local_policy / us_export_policy jars in javahome/jre/lib/security
>> >> more i read, maybe limited by jce.jar
>> >>
>> >> i dont have anything special set for extension jars
>> >>
>> >>
>> >>
>> >>> On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu>
>> wrote:
>> >>>
>> >>> John,
>> >>>
>> >>> What version of the JDK are you running, and what is your extension
>> jars
>> >>> environment variable set to.  Do you have the java cryptology jar
>> >> included
>> >>> (Oracle JDK usually has this, I don't know if Open JDK does).
>> >>>
>> >>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot
>> >> find
>> >>> the java crypto jar used to calculate the given hash.
>> >>>
>> >>> -Michael
>> >>>
>> >>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jr...@vpicu.net>
>> >> wrote:
>> >>>
>> >>>> Hi Lewis,
>> >>>> using the latest docker buggtb/oodt image, which i assume is .8
>> >>>> here’s the command i’m running to test the upload
>> >>>>
>> >>>> filemgr-client --url http://localhost:9000 --operation
>> --ingestProduct
>> >>>> --productName test --productStructure Flat --productTypeName
>> GenericFile
>> >>>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
>> >>>>
>> >>>> i verified that i can upload to the path using the s3 tools on the
>> box /
>> >>>> with same credentials i put in the properties file
>> >>>>
>> >>>> here’s the full exception returned:
>> >>>>
>> >>>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
>> >>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>>       at
>> >>>>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >>>>       at
>> >>>>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>>       at java.lang.reflect.Method.invoke(Method.java:606)
>> >>>>       at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>> >>>>       at
>> >>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>> >>>>       at
>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>> >>>>       at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>> >>>>       at java.lang.Thread.run(Thread.java:745)
>> >>>> Caused by:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
>> >>>>       ... 12 more
>> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>> >>>> request signature: Unable to calculate a request signature: Algorithm
>> >>>> HmacSHA1 not available
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
>> >>>>       at
>> >>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
>> >>>>       at
>> >>>>
>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
>> >>>>       ... 13 more
>> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>> >>>> request signature: Algorithm HmacSHA1 not available
>> >>>>       at
>> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
>> >>>>       ... 20 more
>> >>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1
>> >> not
>> >>>> available
>> >>>>       at javax.crypto.Mac.getInstance(Mac.java:176)
>> >>>>       at
>> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
>> >>>>       ... 21 more
>> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error
>> >>>> ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> >> ]
>> >>>> :
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>>       at
>> >>>>
>> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>> >>>>       at
>> >>>>
>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>> >>>>       at
>> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>> >>>> java.lang.Exception: Failed to ingest product
>> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>> >>>> java.lang.Exception:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error
>> >>>> ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> >> ]
>> >>>> :
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>> >>>>       at
>> >>>>
>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>> >>>>       at
>> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>> >>>> ERROR: Failed to ingest product 'test' : Failed to ingest product
>> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>> >>>> java.lang.Exception:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error
>> >>>> ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> >> ]
>> >>>> :
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>
>> >>>>
>> >>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
>> >>>> lewis.mcgibbney@gmail.com> wrote:
>> >>>>>
>> >>>>> Hi John,
>> >>>>> What beau on of OODT are you using?
>> >>>>> The file types should be encapsulated by the factory implementation
>> I
>> >>>> would
>> >>>>> guess. It should be. Case of mike this the default filemgr store
>> then
>> >> off
>> >>>>> you go.
>> >>>>> Can you provide a paste of how your arriving at your exception
>> please?
>> >>>>> Lewia
>> >>>>>
>> >>>>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net>
>> >> wrote:
>> >>>>>
>> >>>>>> Should S3DataTransferer work with the generic file type from the
>> get
>> >> go
>> >>>> or
>> >>>>>> is there anything i need to change
>> >>>>>> i was using the radix docker but getting 'Caused by:
>> >>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
>> >>>> available’
>> >>>>>> i tried downloading the jce but no dice
>> >>>>>> i might try just building on my local machine and trying again
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>>
>> >>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>> >>>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>> >>>>>>>
>> >>>>>>> Hey John,
>> >>>>>>> You're right, you've nearly answered everything here ;)
>> >>>>>>> OK, so your properties can be located here
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>> >>>>>>> You'll see the usual key, value pairs in there which you can ad
>> >> before
>> >>>>>>> build the project and packaging filemgr.properties along with your
>> >>>>>> compiled
>> >>>>>>> code.
>> >>>>>>> Regarding the following,
>> >>>>>>>
>> >>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <
>> jreynolds@vpicu.net
>> >>>>>> <javascript:;>> wrote:
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> in the near future, i will want to pass in a security token as
>> well
>> >> to
>> >>>>>> the
>> >>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer
>> >> class
>> >>>>>>>> (which expires after an hour), what’s the best practice to read
>> in
>> >>>> this
>> >>>>>>>> transient data from the command line (or wherever else i would
>> >>>> initiate
>> >>>>>> a
>> >>>>>>>> job from) since this wouldnt go in a properties file
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Do you mean that your security token expires hourly? As oppose to
>> >> your
>> >>>> s3
>> >>>>>>> client?
>> >>>>>>> Lets try and defined exactly what you are after here.
>> >>>>>>> Thanks
>> >>>>>>> Lewis
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>> --
>> >>>>> *Lewis*
>> >>>>
>> >>>>
>> >>
>> >>
>>
>>
>

Re: Amazon S3 data transfer for filemanager

Posted by Michael Starch <st...@umich.edu>.
John,

Change:
    -classpath "$FILEMGR_HOME"/lib \
To:
    -classpath "$FILEMGR_HOME/lib/*" \

-Michael

On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <jr...@vpicu.net> wrote:

> Thanks Michael, if i modify the filemgr-client to look like this (at the
> end)
> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
>   -classpath "$FILEMGR_HOME"/lib \
>
> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties
> \
>   -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \
>
> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml
> \
>
> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml
> \
>   org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“
>
> (replacing ext jars with -classpath) then i get
> Error: Could not find or load main class
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
> i assume i’m doing something wrong with classpath but not sure what
>
> > On Mar 12, 2015, at 11:31 AM, Michael Starch <st...@umich.edu> wrote:
> >
> > John,
> >
> > Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
> > there.  If so, it is clobbering the default path for "extension" jars,
> and
> > your java encryption jars are not being picked up. If it does set
> > JAVA_EXT_JARS you have two options:
> >
> > 1. Move all your encryption jars into FILEMGR_HOME/lib/
> > 2. update filemgr-client script to us classpath to specify the jars in
> the
> > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS
> >
> >
> > -Michael
> >
> >
> > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jr...@vpicu.net>
> wrote:
> >
> >> Hi Michael
> >> yeah it’s openjdk 1.7 (“1.7.0_75")
> >> i did download the the unlimited encryption jar from oracle and replaced
> >> the local_policy / us_export_policy jars in javahome/jre/lib/security
> >> more i read, maybe limited by jce.jar
> >>
> >> i dont have anything special set for extension jars
> >>
> >>
> >>
> >>> On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu>
> wrote:
> >>>
> >>> John,
> >>>
> >>> What version of the JDK are you running, and what is your extension
> jars
> >>> environment variable set to.  Do you have the java cryptology jar
> >> included
> >>> (Oracle JDK usually has this, I don't know if Open JDK does).
> >>>
> >>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot
> >> find
> >>> the java crypto jar used to calculate the given hash.
> >>>
> >>> -Michael
> >>>
> >>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jr...@vpicu.net>
> >> wrote:
> >>>
> >>>> Hi Lewis,
> >>>> using the latest docker buggtb/oodt image, which i assume is .8
> >>>> here’s the command i’m running to test the upload
> >>>>
> >>>> filemgr-client --url http://localhost:9000 --operation
> --ingestProduct
> >>>> --productName test --productStructure Flat --productTypeName
> GenericFile
> >>>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
> >>>>
> >>>> i verified that i can upload to the path using the s3 tools on the
> box /
> >>>> with same credentials i put in the properties file
> >>>>
> >>>> here’s the full exception returned:
> >>>>
> >>>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>> Failed to upload product reference /root/test.txt to S3 at
> >>>> usr/src/oodt/data/archive/test/test.txt
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
> >>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>       at
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>>       at
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>       at java.lang.reflect.Method.invoke(Method.java:606)
> >>>>       at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >>>>       at
> >>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >>>>       at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >>>>       at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >>>>       at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >>>>       at
> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >>>>       at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >>>>       at java.lang.Thread.run(Thread.java:745)
> >>>> Caused by:
> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>> Failed to upload product reference /root/test.txt to S3 at
> >>>> usr/src/oodt/data/archive/test/test.txt
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
> >>>>       ... 12 more
> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
> >>>> request signature: Unable to calculate a request signature: Algorithm
> >>>> HmacSHA1 not available
> >>>>       at
> >>>>
> >>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
> >>>>       at
> >>>>
> >>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
> >>>>       at
> >>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
> >>>>       at
> >>>>
> >>
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
> >>>>       at
> >>>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> >>>>       at
> >>>>
> >>
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> >>>>       at
> >>>>
> >>
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
> >>>>       ... 13 more
> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
> >>>> request signature: Algorithm HmacSHA1 not available
> >>>>       at
> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
> >>>>       at
> >>>>
> >>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
> >>>>       ... 20 more
> >>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1
> >> not
> >>>> available
> >>>>       at javax.crypto.Mac.getInstance(Mac.java:176)
> >>>>       at
> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
> >>>>       ... 21 more
> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >>>> ingesting product
> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> >> ]
> >>>> :
> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>> Failed to upload product reference /root/test.txt to S3 at
> >>>> usr/src/oodt/data/archive/test/test.txt
> >>>>       at
> >>>>
> >>
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
> >>>>       at
> >>>>
> >>
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
> >>>>       at
> >>>>
> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> >>>>       at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> >>>>       at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> >>>>       at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
> >>>>       at
> >>>>
> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> >>>>       at
> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> >>>> java.lang.Exception: Failed to ingest product
> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> >>>> java.lang.Exception:
> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >>>> ingesting product
> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> >> ]
> >>>> :
> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>> Failed to upload product reference /root/test.txt to S3 at
> >>>> usr/src/oodt/data/archive/test/test.txt
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
> >>>>       at
> >>>>
> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> >>>>       at
> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> >>>>       at
> >>>>
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> >>>> ERROR: Failed to ingest product 'test' : Failed to ingest product
> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> >>>> java.lang.Exception:
> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >>>> ingesting product
> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> >> ]
> >>>> :
> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >>>> Failed to upload product reference /root/test.txt to S3 at
> >>>> usr/src/oodt/data/archive/test/test.txt
> >>>>
> >>>>
> >>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
> >>>> lewis.mcgibbney@gmail.com> wrote:
> >>>>>
> >>>>> Hi John,
> >>>>> What beau on of OODT are you using?
> >>>>> The file types should be encapsulated by the factory implementation I
> >>>> would
> >>>>> guess. It should be. Case of mike this the default filemgr store then
> >> off
> >>>>> you go.
> >>>>> Can you provide a paste of how your arriving at your exception
> please?
> >>>>> Lewia
> >>>>>
> >>>>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net>
> >> wrote:
> >>>>>
> >>>>>> Should S3DataTransferer work with the generic file type from the get
> >> go
> >>>> or
> >>>>>> is there anything i need to change
> >>>>>> i was using the radix docker but getting 'Caused by:
> >>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
> >>>> available’
> >>>>>> i tried downloading the jce but no dice
> >>>>>> i might try just building on my local machine and trying again
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
> >>>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
> >>>>>>>
> >>>>>>> Hey John,
> >>>>>>> You're right, you've nearly answered everything here ;)
> >>>>>>> OK, so your properties can be located here
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
> >>>>>>> You'll see the usual key, value pairs in there which you can ad
> >> before
> >>>>>>> build the project and packaging filemgr.properties along with your
> >>>>>> compiled
> >>>>>>> code.
> >>>>>>> Regarding the following,
> >>>>>>>
> >>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <
> jreynolds@vpicu.net
> >>>>>> <javascript:;>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> in the near future, i will want to pass in a security token as
> well
> >> to
> >>>>>> the
> >>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer
> >> class
> >>>>>>>> (which expires after an hour), what’s the best practice to read in
> >>>> this
> >>>>>>>> transient data from the command line (or wherever else i would
> >>>> initiate
> >>>>>> a
> >>>>>>>> job from) since this wouldnt go in a properties file
> >>>>>>>
> >>>>>>>
> >>>>>>> Do you mean that your security token expires hourly? As oppose to
> >> your
> >>>> s3
> >>>>>>> client?
> >>>>>>> Lets try and defined exactly what you are after here.
> >>>>>>> Thanks
> >>>>>>> Lewis
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> --
> >>>>> *Lewis*
> >>>>
> >>>>
> >>
> >>
>
>

Re: Amazon S3 data transfer for filemanager

Posted by John Reynolds <jr...@vpicu.net>.
Thanks Michael, if i modify the filemgr-client to look like this (at the end)
"$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
  -classpath "$FILEMGR_HOME"/lib \
  -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties \
  -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \
  -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml \
  -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml \
  org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“

(replacing ext jars with -classpath) then i get
Error: Could not find or load main class org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
i assume i’m doing something wrong with classpath but not sure what

> On Mar 12, 2015, at 11:31 AM, Michael Starch <st...@umich.edu> wrote:
> 
> John,
> 
> Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
> there.  If so, it is clobbering the default path for "extension" jars, and
> your java encryption jars are not being picked up. If it does set
> JAVA_EXT_JARS you have two options:
> 
> 1. Move all your encryption jars into FILEMGR_HOME/lib/
> 2. update filemgr-client script to us classpath to specify the jars in the
> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS
> 
> 
> -Michael
> 
> 
> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jr...@vpicu.net> wrote:
> 
>> Hi Michael
>> yeah it’s openjdk 1.7 (“1.7.0_75")
>> i did download the the unlimited encryption jar from oracle and replaced
>> the local_policy / us_export_policy jars in javahome/jre/lib/security
>> more i read, maybe limited by jce.jar
>> 
>> i dont have anything special set for extension jars
>> 
>> 
>> 
>>> On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu> wrote:
>>> 
>>> John,
>>> 
>>> What version of the JDK are you running, and what is your extension jars
>>> environment variable set to.  Do you have the java cryptology jar
>> included
>>> (Oracle JDK usually has this, I don't know if Open JDK does).
>>> 
>>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot
>> find
>>> the java crypto jar used to calculate the given hash.
>>> 
>>> -Michael
>>> 
>>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jr...@vpicu.net>
>> wrote:
>>> 
>>>> Hi Lewis,
>>>> using the latest docker buggtb/oodt image, which i assume is .8
>>>> here’s the command i’m running to test the upload
>>>> 
>>>> filemgr-client --url http://localhost:9000 --operation --ingestProduct
>>>> --productName test --productStructure Flat --productTypeName GenericFile
>>>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
>>>> 
>>>> i verified that i can upload to the path using the s3 tools on the box /
>>>> with same credentials i put in the properties file
>>>> 
>>>> here’s the full exception returned:
>>>> 
>>>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>> Failed to upload product reference /root/test.txt to S3 at
>>>> usr/src/oodt/data/archive/test/test.txt
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
>>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>       at
>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>       at
>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>       at java.lang.reflect.Method.invoke(Method.java:606)
>>>>       at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>>>>       at
>>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>>>>       at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>>>>       at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>>>>       at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>>>>       at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>>>>       at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>>>>       at java.lang.Thread.run(Thread.java:745)
>>>> Caused by:
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>> Failed to upload product reference /root/test.txt to S3 at
>>>> usr/src/oodt/data/archive/test/test.txt
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
>>>>       ... 12 more
>>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>>>> request signature: Unable to calculate a request signature: Algorithm
>>>> HmacSHA1 not available
>>>>       at
>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
>>>>       at
>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
>>>>       at
>>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
>>>>       at
>>>> 
>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
>>>>       at
>>>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
>>>>       at
>>>> 
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
>>>>       at
>>>> 
>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
>>>>       ... 13 more
>>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>>>> request signature: Algorithm HmacSHA1 not available
>>>>       at
>>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
>>>>       at
>>>> 
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
>>>>       ... 20 more
>>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1
>> not
>>>> available
>>>>       at javax.crypto.Mac.getInstance(Mac.java:176)
>>>>       at
>>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
>>>>       ... 21 more
>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>>>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> ]
>>>> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>> Failed to upload product reference /root/test.txt to S3 at
>>>> usr/src/oodt/data/archive/test/test.txt
>>>>       at
>>>> 
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>>       at
>>>> 
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>>       at
>>>> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>>       at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>       at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>       at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>>>       at
>>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>       at
>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>>>> java.lang.Exception: Failed to ingest product
>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>>>> java.lang.Exception:
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>>>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> ]
>>>> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>> Failed to upload product reference /root/test.txt to S3 at
>>>> usr/src/oodt/data/archive/test/test.txt
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>>>       at
>>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>       at
>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>       at
>>>> 
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>>>> ERROR: Failed to ingest product 'test' : Failed to ingest product
>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>>>> java.lang.Exception:
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>>>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> ]
>>>> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>>>> Failed to upload product reference /root/test.txt to S3 at
>>>> usr/src/oodt/data/archive/test/test.txt
>>>> 
>>>> 
>>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>> 
>>>>> Hi John,
>>>>> What beau on of OODT are you using?
>>>>> The file types should be encapsulated by the factory implementation I
>>>> would
>>>>> guess. It should be. Case of mike this the default filemgr store then
>> off
>>>>> you go.
>>>>> Can you provide a paste of how your arriving at your exception please?
>>>>> Lewia
>>>>> 
>>>>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net>
>> wrote:
>>>>> 
>>>>>> Should S3DataTransferer work with the generic file type from the get
>> go
>>>> or
>>>>>> is there anything i need to change
>>>>>> i was using the radix docker but getting 'Caused by:
>>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
>>>> available’
>>>>>> i tried downloading the jce but no dice
>>>>>> i might try just building on my local machine and trying again
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>>>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>>>>>>> 
>>>>>>> Hey John,
>>>>>>> You're right, you've nearly answered everything here ;)
>>>>>>> OK, so your properties can be located here
>>>>>>> 
>>>>>> 
>>>> 
>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>>>>>>> You'll see the usual key, value pairs in there which you can ad
>> before
>>>>>>> build the project and packaging filemgr.properties along with your
>>>>>> compiled
>>>>>>> code.
>>>>>>> Regarding the following,
>>>>>>> 
>>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jreynolds@vpicu.net
>>>>>> <javascript:;>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> in the near future, i will want to pass in a security token as well
>> to
>>>>>> the
>>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer
>> class
>>>>>>>> (which expires after an hour), what’s the best practice to read in
>>>> this
>>>>>>>> transient data from the command line (or wherever else i would
>>>> initiate
>>>>>> a
>>>>>>>> job from) since this wouldnt go in a properties file
>>>>>>> 
>>>>>>> 
>>>>>>> Do you mean that your security token expires hourly? As oppose to
>> your
>>>> s3
>>>>>>> client?
>>>>>>> Lets try and defined exactly what you are after here.
>>>>>>> Thanks
>>>>>>> Lewis
>>>>>> 
>>>>>> 
>>>>> 
>>>>> --
>>>>> *Lewis*
>>>> 
>>>> 
>> 
>> 


Re: Amazon S3 data transfer for filemanager

Posted by Michael Starch <st...@umich.edu>.
John,

Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
there.  If so, it is clobbering the default path for "extension" jars, and
your java encryption jars are not being picked up. If it does set
JAVA_EXT_JARS you have two options:

1. Move all your encryption jars into FILEMGR_HOME/lib/
2. update filemgr-client script to us classpath to specify the jars in the
FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS


-Michael


On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jr...@vpicu.net> wrote:

> Hi Michael
> yeah it’s openjdk 1.7 (“1.7.0_75")
> i did download the the unlimited encryption jar from oracle and replaced
> the local_policy / us_export_policy jars in javahome/jre/lib/security
> more i read, maybe limited by jce.jar
>
> i dont have anything special set for extension jars
>
>
>
> > On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu> wrote:
> >
> > John,
> >
> > What version of the JDK are you running, and what is your extension jars
> > environment variable set to.  Do you have the java cryptology jar
> included
> > (Oracle JDK usually has this, I don't know if Open JDK does).
> >
> > "Algorithm HmacSHA1 not available" is usually thrown when Java cannot
> find
> > the java crypto jar used to calculate the given hash.
> >
> > -Michael
> >
> > On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jr...@vpicu.net>
> wrote:
> >
> >> Hi Lewis,
> >> using the latest docker buggtb/oodt image, which i assume is .8
> >> here’s the command i’m running to test the upload
> >>
> >> filemgr-client --url http://localhost:9000 --operation --ingestProduct
> >> --productName test --productStructure Flat --productTypeName GenericFile
> >> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
> >>
> >> i verified that i can upload to the path using the s3 tools on the box /
> >> with same credentials i put in the properties file
> >>
> >> here’s the full exception returned:
> >>
> >> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >> Failed to upload product reference /root/test.txt to S3 at
> >> usr/src/oodt/data/archive/test/test.txt
> >>        at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
> >>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>        at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>        at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>        at java.lang.reflect.Method.invoke(Method.java:606)
> >>        at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> >>        at
> >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> >>        at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> >>        at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> >>        at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> >>        at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> >>        at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
> >>        at java.lang.Thread.run(Thread.java:745)
> >> Caused by:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >> Failed to upload product reference /root/test.txt to S3 at
> >> usr/src/oodt/data/archive/test/test.txt
> >>        at
> >>
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
> >>        at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
> >>        ... 12 more
> >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
> >> request signature: Unable to calculate a request signature: Algorithm
> >> HmacSHA1 not available
> >>        at
> >>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
> >>        at
> >>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
> >>        at
> >> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
> >>        at
> >>
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
> >>        at
> >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> >>        at
> >>
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> >>        at
> >>
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
> >>        at
> >>
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
> >>        ... 13 more
> >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
> >> request signature: Algorithm HmacSHA1 not available
> >>        at
> >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
> >>        at
> >>
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
> >>        ... 20 more
> >> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1
> not
> >> available
> >>        at javax.crypto.Mac.getInstance(Mac.java:176)
> >>        at
> >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
> >>        ... 21 more
> >> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> ]
> >> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >> Failed to upload product reference /root/test.txt to S3 at
> >> usr/src/oodt/data/archive/test/test.txt
> >>        at
> >>
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
> >>        at
> >>
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
> >>        at
> >> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> >>        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> >>        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> >>        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> >>        at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
> >>        at
> >>
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
> >>        at
> >> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> >>        at
> >> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> >>        at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> >> java.lang.Exception: Failed to ingest product
> >> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> >> java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> ]
> >> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >> Failed to upload product reference /root/test.txt to S3 at
> >> usr/src/oodt/data/archive/test/test.txt
> >>        at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
> >>        at
> >>
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
> >>        at
> >> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> >>        at
> >> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> >>        at
> >>
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> >> ERROR: Failed to ingest product 'test' : Failed to ingest product
> >> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> >> java.lang.Exception:
> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
> ]
> >> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> >> Failed to upload product reference /root/test.txt to S3 at
> >> usr/src/oodt/data/archive/test/test.txt
> >>
> >>
> >>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>>
> >>> Hi John,
> >>> What beau on of OODT are you using?
> >>> The file types should be encapsulated by the factory implementation I
> >> would
> >>> guess. It should be. Case of mike this the default filemgr store then
> off
> >>> you go.
> >>> Can you provide a paste of how your arriving at your exception please?
> >>> Lewia
> >>>
> >>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net>
> wrote:
> >>>
> >>>> Should S3DataTransferer work with the generic file type from the get
> go
> >> or
> >>>> is there anything i need to change
> >>>> i was using the radix docker but getting 'Caused by:
> >>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
> >> available’
> >>>> i tried downloading the jce but no dice
> >>>> i might try just building on my local machine and trying again
> >>>>
> >>>> Thanks
> >>>>
> >>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
> >>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
> >>>>>
> >>>>> Hey John,
> >>>>> You're right, you've nearly answered everything here ;)
> >>>>> OK, so your properties can be located here
> >>>>>
> >>>>
> >>
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
> >>>>> You'll see the usual key, value pairs in there which you can ad
> before
> >>>>> build the project and packaging filemgr.properties along with your
> >>>> compiled
> >>>>> code.
> >>>>> Regarding the following,
> >>>>>
> >>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jreynolds@vpicu.net
> >>>> <javascript:;>> wrote:
> >>>>>
> >>>>>>
> >>>>>> in the near future, i will want to pass in a security token as well
> to
> >>>> the
> >>>>>> S3 client. to do so i would extend or modify the s3datatransfer
> class
> >>>>>> (which expires after an hour), what’s the best practice to read in
> >> this
> >>>>>> transient data from the command line (or wherever else i would
> >> initiate
> >>>> a
> >>>>>> job from) since this wouldnt go in a properties file
> >>>>>
> >>>>>
> >>>>> Do you mean that your security token expires hourly? As oppose to
> your
> >> s3
> >>>>> client?
> >>>>> Lets try and defined exactly what you are after here.
> >>>>> Thanks
> >>>>> Lewis
> >>>>
> >>>>
> >>>
> >>> --
> >>> *Lewis*
> >>
> >>
>
>

Re: Amazon S3 data transfer for filemanager

Posted by John Reynolds <jr...@vpicu.net>.
Hi Michael
yeah it’s openjdk 1.7 (“1.7.0_75")
i did download the the unlimited encryption jar from oracle and replaced the local_policy / us_export_policy jars in javahome/jre/lib/security
more i read, maybe limited by jce.jar

i dont have anything special set for extension jars 



> On Mar 12, 2015, at 10:35 AM, Michael Starch <st...@umich.edu> wrote:
> 
> John,
> 
> What version of the JDK are you running, and what is your extension jars
> environment variable set to.  Do you have the java cryptology jar included
> (Oracle JDK usually has this, I don't know if Open JDK does).
> 
> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot find
> the java crypto jar used to calculate the given hash.
> 
> -Michael
> 
> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jr...@vpicu.net> wrote:
> 
>> Hi Lewis,
>> using the latest docker buggtb/oodt image, which i assume is .8
>> here’s the command i’m running to test the upload
>> 
>> filemgr-client --url http://localhost:9000 --operation --ingestProduct
>> --productName test --productStructure Flat --productTypeName GenericFile
>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
>> 
>> i verified that i can upload to the path using the s3 tools on the box /
>> with same credentials i put in the properties file
>> 
>> here’s the full exception returned:
>> 
>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> Failed to upload product reference /root/test.txt to S3 at
>> usr/src/oodt/data/archive/test/test.txt
>>        at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:606)
>>        at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>>        at
>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>>        at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>>        at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>>        at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>>        at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>>        at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>>        at java.lang.Thread.run(Thread.java:745)
>> Caused by:
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> Failed to upload product reference /root/test.txt to S3 at
>> usr/src/oodt/data/archive/test/test.txt
>>        at
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
>>        at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
>>        ... 12 more
>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>> request signature: Unable to calculate a request signature: Algorithm
>> HmacSHA1 not available
>>        at
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
>>        at
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
>>        at
>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
>>        at
>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
>>        at
>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
>>        at
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
>>        at
>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
>>        at
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
>>        ... 13 more
>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
>> request signature: Algorithm HmacSHA1 not available
>>        at
>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
>>        at
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
>>        ... 20 more
>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
>> available
>>        at javax.crypto.Mac.getInstance(Mac.java:176)
>>        at
>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
>>        ... 21 more
>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1]
>> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> Failed to upload product reference /root/test.txt to S3 at
>> usr/src/oodt/data/archive/test/test.txt
>>        at
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>        at
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>        at
>> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>        at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
>>        at
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>        at
>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>        at
>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>        at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>> java.lang.Exception: Failed to ingest product
>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>> java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1]
>> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> Failed to upload product reference /root/test.txt to S3 at
>> usr/src/oodt/data/archive/test/test.txt
>>        at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
>>        at
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>>        at
>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>        at
>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>        at
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>> ERROR: Failed to ingest product 'test' : Failed to ingest product
>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>> java.lang.Exception:
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
>> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1]
>> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> Failed to upload product reference /root/test.txt to S3 at
>> usr/src/oodt/data/archive/test/test.txt
>> 
>> 
>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>> 
>>> Hi John,
>>> What beau on of OODT are you using?
>>> The file types should be encapsulated by the factory implementation I
>> would
>>> guess. It should be. Case of mike this the default filemgr store then off
>>> you go.
>>> Can you provide a paste of how your arriving at your exception please?
>>> Lewia
>>> 
>>> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net> wrote:
>>> 
>>>> Should S3DataTransferer work with the generic file type from the get go
>> or
>>>> is there anything i need to change
>>>> i was using the radix docker but getting 'Caused by:
>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
>> available’
>>>> i tried downloading the jce but no dice
>>>> i might try just building on my local machine and trying again
>>>> 
>>>> Thanks
>>>> 
>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>>>>> 
>>>>> Hey John,
>>>>> You're right, you've nearly answered everything here ;)
>>>>> OK, so your properties can be located here
>>>>> 
>>>> 
>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>>>>> You'll see the usual key, value pairs in there which you can ad before
>>>>> build the project and packaging filemgr.properties along with your
>>>> compiled
>>>>> code.
>>>>> Regarding the following,
>>>>> 
>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jreynolds@vpicu.net
>>>> <javascript:;>> wrote:
>>>>> 
>>>>>> 
>>>>>> in the near future, i will want to pass in a security token as well to
>>>> the
>>>>>> S3 client. to do so i would extend or modify the s3datatransfer class
>>>>>> (which expires after an hour), what’s the best practice to read in
>> this
>>>>>> transient data from the command line (or wherever else i would
>> initiate
>>>> a
>>>>>> job from) since this wouldnt go in a properties file
>>>>> 
>>>>> 
>>>>> Do you mean that your security token expires hourly? As oppose to your
>> s3
>>>>> client?
>>>>> Lets try and defined exactly what you are after here.
>>>>> Thanks
>>>>> Lewis
>>>> 
>>>> 
>>> 
>>> --
>>> *Lewis*
>> 
>> 


Re: Amazon S3 data transfer for filemanager

Posted by Michael Starch <st...@umich.edu>.
John,

What version of the JDK are you running, and what is your extension jars
environment variable set to.  Do you have the java cryptology jar included
(Oracle JDK usually has this, I don't know if Open JDK does).

"Algorithm HmacSHA1 not available" is usually thrown when Java cannot find
the java crypto jar used to calculate the given hash.

-Michael

On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jr...@vpicu.net> wrote:

> Hi Lewis,
> using the latest docker buggtb/oodt image, which i assume is .8
> here’s the command i’m running to test the upload
>
> filemgr-client --url http://localhost:9000 --operation --ingestProduct
> --productName test --productStructure Flat --productTypeName GenericFile
> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
>
> i verified that i can upload to the path using the s3 tools on the box /
> with same credentials i put in the properties file
>
> here’s the full exception returned:
>
> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> Failed to upload product reference /root/test.txt to S3 at
> usr/src/oodt/data/archive/test/test.txt
>         at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>         at
> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>         at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>         at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>         at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>         at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>         at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> Failed to upload product reference /root/test.txt to S3 at
> usr/src/oodt/data/archive/test/test.txt
>         at
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
>         at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
>         ... 12 more
> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
> request signature: Unable to calculate a request signature: Algorithm
> HmacSHA1 not available
>         at
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
>         at
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
>         at
> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
>         at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
>         at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
>         at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
>         at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
>         at
> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
>         ... 13 more
> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a
> request signature: Algorithm HmacSHA1 not available
>         at
> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
>         at
> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
>         ... 20 more
> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
> available
>         at javax.crypto.Mac.getInstance(Mac.java:176)
>         at
> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
>         ... 21 more
> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1]
> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> Failed to upload product reference /root/test.txt to S3 at
> usr/src/oodt/data/archive/test/test.txt
>         at
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>         at
> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>         at
> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>         at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>         at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>         at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>         at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
>         at
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>         at
> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>         at
> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>         at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> java.lang.Exception: Failed to ingest product
> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1]
> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> Failed to upload product reference /root/test.txt to S3 at
> usr/src/oodt/data/archive/test/test.txt
>         at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
>         at
> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>         at
> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>         at
> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>         at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
> ERROR: Failed to ingest product 'test' : Failed to ingest product
> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
> java.lang.Exception:
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1]
> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
> Failed to upload product reference /root/test.txt to S3 at
> usr/src/oodt/data/archive/test/test.txt
>
>
> > On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
> >
> > Hi John,
> > What beau on of OODT are you using?
> > The file types should be encapsulated by the factory implementation I
> would
> > guess. It should be. Case of mike this the default filemgr store then off
> > you go.
> > Can you provide a paste of how your arriving at your exception please?
> > Lewia
> >
> > On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net> wrote:
> >
> >> Should S3DataTransferer work with the generic file type from the get go
> or
> >> is there anything i need to change
> >> i was using the radix docker but getting 'Caused by:
> >> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not
> available’
> >> i tried downloading the jce but no dice
> >> i might try just building on my local machine and trying again
> >>
> >> Thanks
> >>
> >>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
> >>>
> >>> Hey John,
> >>> You're right, you've nearly answered everything here ;)
> >>> OK, so your properties can be located here
> >>>
> >>
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
> >>> You'll see the usual key, value pairs in there which you can ad before
> >>> build the project and packaging filemgr.properties along with your
> >> compiled
> >>> code.
> >>> Regarding the following,
> >>>
> >>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jreynolds@vpicu.net
> >> <javascript:;>> wrote:
> >>>
> >>>>
> >>>> in the near future, i will want to pass in a security token as well to
> >> the
> >>>> S3 client. to do so i would extend or modify the s3datatransfer class
> >>>> (which expires after an hour), what’s the best practice to read in
> this
> >>>> transient data from the command line (or wherever else i would
> initiate
> >> a
> >>>> job from) since this wouldnt go in a properties file
> >>>
> >>>
> >>> Do you mean that your security token expires hourly? As oppose to your
> s3
> >>> client?
> >>> Lets try and defined exactly what you are after here.
> >>> Thanks
> >>> Lewis
> >>
> >>
> >
> > --
> > *Lewis*
>
>

Re: Amazon S3 data transfer for filemanager

Posted by John Reynolds <jr...@vpicu.net>.
Hi Lewis,
using the latest docker buggtb/oodt image, which i assume is .8
here’s the command i’m running to test the upload

filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName test --productStructure Flat --productTypeName GenericFile --metadataFile file:///root/test.txt.met --refs file:///root/test.txt

i verified that i can upload to the path using the s3 tools on the box / with same credentials i put in the properties file

here’s the full exception returned:

rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
	at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
	at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
	at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
	at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
	at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
	at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
	... 12 more
Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Algorithm HmacSHA1 not available
	at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
	at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
	at com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
	at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
	... 13 more
Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Algorithm HmacSHA1 not available
	at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
	at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
	... 20 more
Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available
	at javax.crypto.Mac.getInstance(Mac.java:176)
	at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
	... 21 more
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
	at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
	at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
	at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
	at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
	at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
	at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
	at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
	at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
java.lang.Exception: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
	at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
	at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
	at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
ERROR: Failed to ingest product 'test' : Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt


> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <le...@gmail.com> wrote:
> 
> Hi John,
> What beau on of OODT are you using?
> The file types should be encapsulated by the factory implementation I would
> guess. It should be. Case of mike this the default filemgr store then off
> you go.
> Can you provide a paste of how your arriving at your exception please?
> Lewia
> 
> On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net> wrote:
> 
>> Should S3DataTransferer work with the generic file type from the get go or
>> is there anything i need to change
>> i was using the radix docker but getting 'Caused by:
>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available’
>> i tried downloading the jce but no dice
>> i might try just building on my local machine and trying again
>> 
>> Thanks
>> 
>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>>> 
>>> Hey John,
>>> You're right, you've nearly answered everything here ;)
>>> OK, so your properties can be located here
>>> 
>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>>> You'll see the usual key, value pairs in there which you can ad before
>>> build the project and packaging filemgr.properties along with your
>> compiled
>>> code.
>>> Regarding the following,
>>> 
>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jreynolds@vpicu.net
>> <javascript:;>> wrote:
>>> 
>>>> 
>>>> in the near future, i will want to pass in a security token as well to
>> the
>>>> S3 client. to do so i would extend or modify the s3datatransfer class
>>>> (which expires after an hour), what’s the best practice to read in this
>>>> transient data from the command line (or wherever else i would initiate
>> a
>>>> job from) since this wouldnt go in a properties file
>>> 
>>> 
>>> Do you mean that your security token expires hourly? As oppose to your s3
>>> client?
>>> Lets try and defined exactly what you are after here.
>>> Thanks
>>> Lewis
>> 
>> 
> 
> -- 
> *Lewis*


Re: Amazon S3 data transfer for filemanager

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi John,
What beau on of OODT are you using?
The file types should be encapsulated by the factory implementation I would
guess. It should be. Case of mike this the default filemgr store then off
you go.
Can you provide a paste of how your arriving at your exception please?
Lewia

On Wednesday, March 11, 2015, John Reynolds <jr...@vpicu.net> wrote:

> Should S3DataTransferer work with the generic file type from the get go or
> is there anything i need to change
> i was using the radix docker but getting 'Caused by:
> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available’
> i tried downloading the jce but no dice
> i might try just building on my local machine and trying again
>
> Thanks
>
> > On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
> >
> > Hey John,
> > You're right, you've nearly answered everything here ;)
> > OK, so your properties can be located here
> >
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
> > You'll see the usual key, value pairs in there which you can ad before
> > build the project and packaging filemgr.properties along with your
> compiled
> > code.
> > Regarding the following,
> >
> > On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jreynolds@vpicu.net
> <javascript:;>> wrote:
> >
> >>
> >> in the near future, i will want to pass in a security token as well to
> the
> >> S3 client. to do so i would extend or modify the s3datatransfer class
> >> (which expires after an hour), what’s the best practice to read in this
> >> transient data from the command line (or wherever else i would initiate
> a
> >> job from) since this wouldnt go in a properties file
> >
> >
> > Do you mean that your security token expires hourly? As oppose to your s3
> > client?
> > Lets try and defined exactly what you are after here.
> > Thanks
> > Lewis
>
>

-- 
*Lewis*

Re: Amazon S3 data transfer for filemanager

Posted by John Reynolds <jr...@vpicu.net>.
Should S3DataTransferer work with the generic file type from the get go or is there anything i need to change
i was using the radix docker but getting 'Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available’
i tried downloading the jce but no dice
i might try just building on my local machine and trying again

Thanks

> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <le...@gmail.com> wrote:
> 
> Hey John,
> You're right, you've nearly answered everything here ;)
> OK, so your properties can be located here
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
> You'll see the usual key, value pairs in there which you can ad before
> build the project and packaging filemgr.properties along with your compiled
> code.
> Regarding the following,
> 
> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jr...@vpicu.net> wrote:
> 
>> 
>> in the near future, i will want to pass in a security token as well to the
>> S3 client. to do so i would extend or modify the s3datatransfer class
>> (which expires after an hour), what’s the best practice to read in this
>> transient data from the command line (or wherever else i would initiate a
>> job from) since this wouldnt go in a properties file
> 
> 
> Do you mean that your security token expires hourly? As oppose to your s3
> client?
> Lets try and defined exactly what you are after here.
> Thanks
> Lewis


Re: Amazon S3 data transfer for filemanager

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hey John,
You're right, you've nearly answered everything here ;)
OK, so your properties can be located here
https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
You'll see the usual key, value pairs in there which you can ad before
build the project and packaging filemgr.properties along with your compiled
code.
Regarding the following,

On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jr...@vpicu.net> wrote:

>
> in the near future, i will want to pass in a security token as well to the
> S3 client. to do so i would extend or modify the s3datatransfer class
> (which expires after an hour), what’s the best practice to read in this
> transient data from the command line (or wherever else i would initiate a
> job from) since this wouldnt go in a properties file


Do you mean that your security token expires hourly? As oppose to your s3
client?
Lets try and defined exactly what you are after here.
Thanks
Lewis

Re: Amazon S3 data transfer for filemanager

Posted by John Reynolds <jr...@vpicu.net>.
Ok, i think i’m exploring / answering my own questions :)
I found the implementation of S3DataTransferer / Factory and it looks like what i need to start
in the near future, i will want to pass in a security token as well to the S3 client. to do so i would extend or modify the s3datatransfer class
(which expires after an hour), what’s the best practice to read in this transient data from the command line (or wherever else i would initiate a job from) since this wouldnt go in a properties file


> On Mar 10, 2015, at 3:48 PM, John Reynolds <jr...@vpicu.net> wrote:
> 
> Hi,
> i’m new to the terminology of OODT (and OODT) so please bear with me.
> i’m looking to do a couple things with OODT and integration with AWS, and wanted to see if anyone has done this already.
> (I searched through the mail archive, and the wiki, and havent found quite what i’m looking for, other than some references to an S3 Datatransferer, that may or may not be complete)
> 
> So, what i’m looking to do:
> 1. ingest files into oodt file manager with S3 as the source
> 2. store files in S3 that have been ingested by file manager (from an arbitrary source)
> 
> what would be the general strategy for doing each of those things (what classes need to be implemented / extended)
> or if someone has done this already, let me know
> 
> Thanks!
> John Reynolds
> 
>