Using perl client library with https

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Using perl client library with https

joe carlson
New problem.

We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.

I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:

FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
string=><model name="genomic" package="org.intermine.model.bio">
<class name="DatabaseReference" is-interface="true">
        <attribute name="identifier" type="java.lang.String"/>
        <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
        <reference name="source" referenced-type="DataSource”/>
….

This went well and it appears to use the https connection retrieve the contents.


CONNECTING TO phytozome.jgi.doe.gov:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
UNENCODED-PARAM: op1==
UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
UNENCODED-PARAM: format=json
UNENCODED-PARAM: name=GO_terms_for_genes
UNENCODED-PARAM: value1=Potri.014G066100
UNENCODED-PARAM: code1=A
SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Webservice::InterMine-1.0405/Perl client library
Content-Length: 110

op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.

The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.

The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.

Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 

For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.

Thanks,

Joe Carlson


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Using perl client library with https

joe carlson
For what it’s worth, I’ve verified that if I add NetHTTPS as a type in Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS and open the connection with Net::HTTPS, then I can use the perl API over https.

But I don’t know enough about Moose to adapt the code to work with either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call $self->set_connection($connection) which checks that this is a specific class Net::HTTP. Is there a way to get Moose to accept alternative types in the type checking?

Joe


On Feb 17, 2016, at 10:15 AM, Joe Carlson <[hidden email]> wrote:

New problem.

We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.

I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:

FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
string=><model name="genomic" package="org.intermine.model.bio">
<class name="DatabaseReference" is-interface="true">
        <attribute name="identifier" type="java.lang.String"/>
        <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
        <reference name="source" referenced-type="DataSource”/>
….

This went well and it appears to use the https connection retrieve the contents.


CONNECTING TO phytozome.jgi.doe.gov:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
UNENCODED-PARAM: op1==
UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
UNENCODED-PARAM: format=json
UNENCODED-PARAM: name=GO_terms_for_genes
UNENCODED-PARAM: value1=Potri.014G066100
UNENCODED-PARAM: code1=A
SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Webservice::InterMine-1.0405/Perl client library
Content-Length: 110

op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.

The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.

The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.

Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 

For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.

Thanks,

Joe Carlson



_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Using perl client library with https

Josh Goodman
In reply to this post by joe carlson
Hi Joe,

You can use type unions for this.  Given how this has been implemented here, I would do the following.

Add a new type for NetHTTPS in Webservice/InterMine/Types.pm:

class_type NetHTTPS, { class => 'Net::HTTPS', };

Then modify ResultIterator to import the new type and use a type union on the connection attribute.

use Webservice::InterMine::Types qw(Uri HTTPCode NetHTTP NetHTTPS RowFormat JsonFormat RequestFormat RowParser);
...
has connection => (
    is        => 'ro',
    isa       => 'NetHTTP|NetHTTPS',
    writer    => 'set_connection',
    predicate => 'has_connection',
);

Josh

On Wed, Feb 17, 2016 at 2:10 PM Joe Carlson <[hidden email]> wrote:
For what it’s worth, I’ve verified that if I add NetHTTPS as a type in Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS and open the connection with Net::HTTPS, then I can use the perl API over https.

But I don’t know enough about Moose to adapt the code to work with either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call $self->set_connection($connection) which checks that this is a specific class Net::HTTP. Is there a way to get Moose to accept alternative types in the type checking?

Joe


On Feb 17, 2016, at 10:15 AM, Joe Carlson <[hidden email]> wrote:

New problem.

We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.

I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:

FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
string=><model name="genomic" package="org.intermine.model.bio">
<class name="DatabaseReference" is-interface="true">
        <attribute name="identifier" type="java.lang.String"/>
        <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
        <reference name="source" referenced-type="DataSource”/>
….

This went well and it appears to use the https connection retrieve the contents.


CONNECTING TO phytozome.jgi.doe.gov:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
UNENCODED-PARAM: op1==
UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
UNENCODED-PARAM: format=json
UNENCODED-PARAM: name=GO_terms_for_genes
UNENCODED-PARAM: value1=Potri.014G066100
UNENCODED-PARAM: code1=A
SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Webservice::InterMine-1.0405/Perl client library
Content-Length: 110

op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.

The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.

The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.

Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 

For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.

Thanks,

Joe Carlson



_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Using perl client library with https

joe carlson
Ah! I had added the class_type to Types.pm and I tried isa=>[NetHTTP,NetHTTPS]. That didn’t work. I didnt know about the pipe symbol.

But your syntax does not work for me with the quotes in there. I have more success if I write it as

isa => NetHTTP|NetHTTPS,

and it even work when I query flymine!

Are you folks the maintainers of the module in CPAN? Can you push out a update? 

thanks,

Joe

On Feb 17, 2016, at 12:12 PM, Josh Goodman <[hidden email]> wrote:

Hi Joe,

You can use type unions for this.  Given how this has been implemented here, I would do the following.

Add a new type for NetHTTPS in Webservice/InterMine/Types.pm:

class_type NetHTTPS, { class => 'Net::HTTPS', };

Then modify ResultIterator to import the new type and use a type union on the connection attribute.

use Webservice::InterMine::Types qw(Uri HTTPCode NetHTTP NetHTTPS RowFormat JsonFormat RequestFormat RowParser);
...
has connection => (
    is        => 'ro',
    isa       => 'NetHTTP|NetHTTPS',
    writer    => 'set_connection',
    predicate => 'has_connection',
);

Josh

On Wed, Feb 17, 2016 at 2:10 PM Joe Carlson <[hidden email]> wrote:
For what it’s worth, I’ve verified that if I add NetHTTPS as a type in Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS and open the connection with Net::HTTPS, then I can use the perl API over https.

But I don’t know enough about Moose to adapt the code to work with either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call $self->set_connection($connection) which checks that this is a specific class Net::HTTP. Is there a way to get Moose to accept alternative types in the type checking?

Joe


On Feb 17, 2016, at 10:15 AM, Joe Carlson <[hidden email]> wrote:

New problem.

We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.

I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:

FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
string=><model name="genomic" package="org.intermine.model.bio">
<class name="DatabaseReference" is-interface="true">
        <attribute name="identifier" type="java.lang.String"/>
        <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
        <reference name="source" referenced-type="DataSource”/>
….

This went well and it appears to use the https connection retrieve the contents.


CONNECTING TO phytozome.jgi.doe.gov:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
UNENCODED-PARAM: op1==
UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
UNENCODED-PARAM: format=json
UNENCODED-PARAM: name=GO_terms_for_genes
UNENCODED-PARAM: value1=Potri.014G066100
UNENCODED-PARAM: code1=A
SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Webservice::InterMine-1.0405/Perl client library
Content-Length: 110

op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.

The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.

The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.

Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 

For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.

Thanks,

Joe Carlson




_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Using perl client library with https

Josh Goodman
In reply to this post by Josh Goodman
Hi Joe,

Apologies for the incorrect syntax, I mixed up Moose vs MooseX types.  I am not a maintainer of the CPAN module, but I created a pull request so that one of the InterMine devs can take a look.


Cheers,
Josh

On Wed, Feb 17, 2016 at 3:57 PM Joe Carlson <[hidden email]> wrote:
Ah! I had added the class_type to Types.pm and I tried isa=>[NetHTTP,NetHTTPS]. That didn’t work. I didnt know about the pipe symbol.

But your syntax does not work for me with the quotes in there. I have more success if I write it as

isa => NetHTTP|NetHTTPS,

and it even work when I query flymine!

Are you folks the maintainers of the module in CPAN? Can you push out a update? 

thanks,

Joe

On Feb 17, 2016, at 12:12 PM, Josh Goodman <[hidden email]> wrote:

Hi Joe,

You can use type unions for this.  Given how this has been implemented here, I would do the following.

Add a new type for NetHTTPS in Webservice/InterMine/Types.pm:

class_type NetHTTPS, { class => 'Net::HTTPS', };

Then modify ResultIterator to import the new type and use a type union on the connection attribute.

use Webservice::InterMine::Types qw(Uri HTTPCode NetHTTP NetHTTPS RowFormat JsonFormat RequestFormat RowParser);
...
has connection => (
    is        => 'ro',
    isa       => 'NetHTTP|NetHTTPS',
    writer    => 'set_connection',
    predicate => 'has_connection',
);

Josh

On Wed, Feb 17, 2016 at 2:10 PM Joe Carlson <[hidden email]> wrote:
For what it’s worth, I’ve verified that if I add NetHTTPS as a type in Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS and open the connection with Net::HTTPS, then I can use the perl API over https.

But I don’t know enough about Moose to adapt the code to work with either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call $self->set_connection($connection) which checks that this is a specific class Net::HTTP. Is there a way to get Moose to accept alternative types in the type checking?

Joe


On Feb 17, 2016, at 10:15 AM, Joe Carlson <[hidden email]> wrote:

New problem.

We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.

I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:

FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
string=><model name="genomic" package="org.intermine.model.bio">
<class name="DatabaseReference" is-interface="true">
        <attribute name="identifier" type="java.lang.String"/>
        <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
        <reference name="source" referenced-type="DataSource”/>
….

This went well and it appears to use the https connection retrieve the contents.


CONNECTING TO phytozome.jgi.doe.gov:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
UNENCODED-PARAM: op1==
UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
UNENCODED-PARAM: format=json
UNENCODED-PARAM: name=GO_terms_for_genes
UNENCODED-PARAM: value1=Potri.014G066100
UNENCODED-PARAM: code1=A
SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Webservice::InterMine-1.0405/Perl client library
Content-Length: 110

op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.

The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.

The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.

Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 

For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.

Thanks,

Joe Carlson




_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Using perl client library with https

joe carlson

On Feb 17, 2016, at 9:19 PM, Josh Goodman <[hidden email]> wrote:

Hi Joe,

Apologies for the incorrect syntax, I mixed up Moose vs MooseX types.  I am not a maintainer of the CPAN module, but I created a pull request so that one of the InterMine devs can take a look.



yep. this was pretty much what I did, too.

Motivated by this issue, I checked the other API’s to see if they had any trouble with https. Java and python passed my “exhaustive" testing (1 script each).

But Ruby failed. When I tried my first script, it sent me a nastygram:


> ruby template.rb 
/Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:173:in `rescue in initialize': Error fetching version at https://phytozome.jgi.doe.gov/phytomine/service/version: wrong status line: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">" (InterMine::ServiceError)
from /Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:170:in `initialize'
from template.rb:16:in `new'
from template.rb:16:in `<main>'

I know absolutely no ruby. And I’m not even certain that this is a https issue. But it looks like it’s getting some unexpected html when requesting the version. Should this be checked by some rubyist?

Thanks for all your help,

Joe

Cheers,
Josh

On Wed, Feb 17, 2016 at 3:57 PM Joe Carlson <[hidden email]> wrote:
Ah! I had added the class_type to Types.pm and I tried isa=>[NetHTTP,NetHTTPS]. That didn’t work. I didnt know about the pipe symbol.

But your syntax does not work for me with the quotes in there. I have more success if I write it as

isa => NetHTTP|NetHTTPS,

and it even work when I query flymine!

Are you folks the maintainers of the module in CPAN? Can you push out a update? 

thanks,

Joe

On Feb 17, 2016, at 12:12 PM, Josh Goodman <[hidden email]> wrote:

Hi Joe,

You can use type unions for this.  Given how this has been implemented here, I would do the following.

Add a new type for NetHTTPS in Webservice/InterMine/Types.pm:

class_type NetHTTPS, { class => 'Net::HTTPS', };

Then modify ResultIterator to import the new type and use a type union on the connection attribute.

use Webservice::InterMine::Types qw(Uri HTTPCode NetHTTP NetHTTPS RowFormat JsonFormat RequestFormat RowParser);
...
has connection => (
    is        => 'ro',
    isa       => 'NetHTTP|NetHTTPS',
    writer    => 'set_connection',
    predicate => 'has_connection',
);

Josh

On Wed, Feb 17, 2016 at 2:10 PM Joe Carlson <[hidden email]> wrote:
For what it’s worth, I’ve verified that if I add NetHTTPS as a type in Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS and open the connection with Net::HTTPS, then I can use the perl API over https.

But I don’t know enough about Moose to adapt the code to work with either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call $self->set_connection($connection) which checks that this is a specific class Net::HTTP. Is there a way to get Moose to accept alternative types in the type checking?

Joe


On Feb 17, 2016, at 10:15 AM, Joe Carlson <[hidden email]> wrote:

New problem.

We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.

I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:

FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
string=><model name="genomic" package="org.intermine.model.bio">
<class name="DatabaseReference" is-interface="true">
        <attribute name="identifier" type="java.lang.String"/>
        <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
        <reference name="source" referenced-type="DataSource”/>
….

This went well and it appears to use the https connection retrieve the contents.


CONNECTING TO phytozome.jgi.doe.gov:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
UNENCODED-PARAM: op1==
UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
UNENCODED-PARAM: format=json
UNENCODED-PARAM: name=GO_terms_for_genes
UNENCODED-PARAM: value1=Potri.014G066100
UNENCODED-PARAM: code1=A
SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Webservice::InterMine-1.0405/Perl client library
Content-Length: 110

op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.

The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.

The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.

Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 

For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.

Thanks,

Joe Carlson





_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Using perl client library with https

Josh Goodman
In reply to this post by Josh Goodman
Yes, I haven't a clue about Ruby.

Josh

On Thu, Feb 18, 2016 at 1:06 AM Joe Carlson <[hidden email]> wrote:
On Feb 17, 2016, at 9:19 PM, Josh Goodman <[hidden email]> wrote:

Hi Joe,

Apologies for the incorrect syntax, I mixed up Moose vs MooseX types.  I am not a maintainer of the CPAN module, but I created a pull request so that one of the InterMine devs can take a look.



yep. this was pretty much what I did, too.

Motivated by this issue, I checked the other API’s to see if they had any trouble with https. Java and python passed my “exhaustive" testing (1 script each).

But Ruby failed. When I tried my first script, it sent me a nastygram:


> ruby template.rb 
/Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:173:in `rescue in initialize': Error fetching version at https://phytozome.jgi.doe.gov/phytomine/service/version: wrong status line: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">" (InterMine::ServiceError)
from /Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:170:in `initialize'
from template.rb:16:in `new'
from template.rb:16:in `<main>'

I know absolutely no ruby. And I’m not even certain that this is a https issue. But it looks like it’s getting some unexpected html when requesting the version. Should this be checked by some rubyist?

Thanks for all your help,

Joe

Cheers,
Josh

On Wed, Feb 17, 2016 at 3:57 PM Joe Carlson <[hidden email]> wrote:
Ah! I had added the class_type to Types.pm and I tried isa=>[NetHTTP,NetHTTPS]. That didn’t work. I didnt know about the pipe symbol.

But your syntax does not work for me with the quotes in there. I have more success if I write it as

isa => NetHTTP|NetHTTPS,

and it even work when I query flymine!

Are you folks the maintainers of the module in CPAN? Can you push out a update? 

thanks,

Joe

On Feb 17, 2016, at 12:12 PM, Josh Goodman <[hidden email]> wrote:

Hi Joe,

You can use type unions for this.  Given how this has been implemented here, I would do the following.

Add a new type for NetHTTPS in Webservice/InterMine/Types.pm:

class_type NetHTTPS, { class => 'Net::HTTPS', };

Then modify ResultIterator to import the new type and use a type union on the connection attribute.

use Webservice::InterMine::Types qw(Uri HTTPCode NetHTTP NetHTTPS RowFormat JsonFormat RequestFormat RowParser);
...
has connection => (
    is        => 'ro',
    isa       => 'NetHTTP|NetHTTPS',
    writer    => 'set_connection',
    predicate => 'has_connection',
);

Josh

On Wed, Feb 17, 2016 at 2:10 PM Joe Carlson <[hidden email]> wrote:
For what it’s worth, I’ve verified that if I add NetHTTPS as a type in Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS and open the connection with Net::HTTPS, then I can use the perl API over https.

But I don’t know enough about Moose to adapt the code to work with either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call $self->set_connection($connection) which checks that this is a specific class Net::HTTP. Is there a way to get Moose to accept alternative types in the type checking?

Joe


On Feb 17, 2016, at 10:15 AM, Joe Carlson <[hidden email]> wrote:

New problem.

We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.

I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:

FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
string=><model name="genomic" package="org.intermine.model.bio">
<class name="DatabaseReference" is-interface="true">
        <attribute name="identifier" type="java.lang.String"/>
        <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
        <reference name="source" referenced-type="DataSource”/>
….

This went well and it appears to use the https connection retrieve the contents.


CONNECTING TO phytozome.jgi.doe.gov:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
UNENCODED-PARAM: op1==
UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
UNENCODED-PARAM: format=json
UNENCODED-PARAM: name=GO_terms_for_genes
UNENCODED-PARAM: value1=Potri.014G066100
UNENCODED-PARAM: code1=A
SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Webservice::InterMine-1.0405/Perl client library
Content-Length: 110

op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.

The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.

The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.

Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 

For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.

Thanks,

Joe Carlson





_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev