runtime error

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

runtime error

Joel Richardson-2

Hi devs,

From time to time, MouseMine gets into a state where users are unable to create lists or log in. The error log says:

java.lang.RuntimeException: Unable to load user profileat org.intermine.api.profile.TagManager.getUserProfile(TagManager.java:626)at org.intermine.api.profile.TagManager.checkUserExists(TagManager.java:93)at org.intermine.api.profile.TagManager.getTags(TagManager.java:386)at org.intermine.api.template.TemplateManager.getTemplatesWithTag(TemplateManager.java:381)at org.intermine.api.template.TemplateManager.getTemplatesWithTag(TemplateManager.java:363)at org.intermine.api.template.TemplateManager.getConversionTemplates(TemplateManager.java:347)at org.intermine.web.struts.ConvertBagController.execute(ConvertBagController.java:65)at 
etc...

Restarting tomcat seems to clear the problem.
Any ideas what might be going on?

Thanks,
Joel

-- 
Joel E. Richardson, Ph.D.
Sr. Research Scientist
Mouse Genome Informatics
The Jackson Laboratory
600 Main Street
Bar Harbor, Maine 04609
207-288-6435

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: runtime error

Alex Kalderimis-2
I suspect this might have something to do with DB connections, which may have been closed to the user profile at some point. I'll look at the error message, which is rather non-specific and unhelpful.

Alex

On 15 Jul 2014 15:48, Joel Richardson <[hidden email]> wrote:

>
>
> Hi devs,
>
> From time to time, MouseMine gets into a state where users are unable to create lists or log in. The error log says:
>
> java.lang.RuntimeException: Unable to load user profileat org.intermine.api.profile.TagManager.getUserProfile(TagManager.java:626)at org.intermine.api.profile.TagManager.checkUserExists(TagManager.java:93)at org.intermine.api.profile.TagManager.getTags(TagManager.java:386)at org.intermine.api.template.TemplateManager.getTemplatesWithTag(TemplateManager.java:381)at org.intermine.api.template.TemplateManager.getTemplatesWithTag(TemplateManager.java:363)at org.intermine.api.template.TemplateManager.getConversionTemplates(TemplateManager.java:347)at org.intermine.web.struts.ConvertBagController.execute(ConvertBagController.java:65)at 
> etc...
>
> Restarting tomcat seems to clear the problem.
> Any ideas what might be going on?
>
> Thanks,
> Joel
>
> -- 
> Joel E. Richardson, Ph.D.
> Sr. Research Scientist
> Mouse Genome Informatics
> The Jackson Laboratory
> 600 Main Street
> Bar Harbor, Maine 04609
> 207-288-6435
> [hidden email]
>
> The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: runtime error

vkrishna
Hi Joel,

We actually encountered this exact same issue when we were initially getting ThaleMine (our InterMine interface to Arabidopsis thaliana genome data, https://apps.araport.org/thalemine/) up and running on our production systems.
Similar to your observation, we also did notice that this issue resolved itself temporarily after a Tomcat restart.

After spending a lot of time troubleshooting the issue with the help of the InterMine developers and our in-house IT staff, we arrived at the following conclusion:
We believe the cause of the error (red rectangles of death or RROD) was JCVI software that closed connections between servers sitting on different subnets.
We were able to demonstrate days of uptime without any RROD, merely by using a web server and database server that were on the same subnet (though there were other variables that still need to be ruled out).
We still have the outstanding issue that InterMine software, in particular its connection pool, never recovers from a lost connection.

Hope this provides you with a fresh perspective into this particular issue.

Thank you!

Best Regards,
Vivek

On Behalf of the Araport Team

--
Vivek Krishnakumar
Sr. Bioinformatics Engineer
Plant Genomics

J. Craig Venter Institute
Phone: 301-795-7363
Fax: 301-795-7070


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: runtime error

Joel Richardson-2

Thanks Vivek. I don't know if this applies in our situation because we have tomcat and postgres running of the same server. But maybe the issue creeps in some other way I'm not aware of. Anyway, you've given us something to look into. Thanks again.

Joel
-- 
Joel E. Richardson, Ph.D.
Sr. Research Scientist
Mouse Genome Informatics
The Jackson Laboratory
600 Main Street
Bar Harbor, Maine 04609
207-288-6435

From: <Krishnakumar>, Vivek <[hidden email]>
Date: Tuesday, July 15, 2014 12:21 PM
To: Joel Richardson <[hidden email]>
Cc: "[hidden email]" <[hidden email]>, Intermine Developer List <[hidden email]>, DL AIP <[hidden email]>
Subject: Re: [InterMine Dev] runtime error

Hi Joel,

We actually encountered this exact same issue when we were initially getting ThaleMine (our InterMine interface to Arabidopsis thaliana genome data, https://apps.araport.org/thalemine/) up and running on our production systems.
Similar to your observation, we also did notice that this issue resolved itself temporarily after a Tomcat restart.

After spending a lot of time troubleshooting the issue with the help of the InterMine developers and our in-house IT staff, we arrived at the following conclusion:
We believe the cause of the error (red rectangles of death or RROD) was JCVI software that closed connections between servers sitting on different subnets.
We were able to demonstrate days of uptime without any RROD, merely by using a web server and database server that were on the same subnet (though there were other variables that still need to be ruled out).
We still have the outstanding issue that InterMine software, in particular its connection pool, never recovers from a lost connection.

Hope this provides you with a fresh perspective into this particular issue.

Thank you!

Best Regards,
Vivek

On Behalf of the Araport Team

--
Vivek Krishnakumar
Sr. Bioinformatics Engineer
Plant Genomics

J. Craig Venter Institute
Phone: 301-795-7363
Fax: 301-795-7070

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: runtime error

vkrishna
Hi Joel,

One more thing to consider is that there is ongoing work by the InterMine team to replace the existing JDBC library with a new one, called Hikari Connection Pool library (HikariCP), which should mitigate such issues of dropped/stale connections in the future.
Here is the related GitHub issue: https://github.com/intermine/intermine/issues/558
and the relevant pull request: https://github.com/intermine/intermine/pull/699
All these changes should be in the “beta” branch of the upstream/intermine repository soon.

And there are teams at Cambridge and specific members in our team at JCVI, who are independently testing out this new connection pool library (the Phytozome group at JGI have showed interest in this and are probably testing things out at their end as well).
From some (very) preliminary testing at our end, everything seems to work fine but because of the new library, there seems to be a drastic increase (almost three-fold) in load times (we are unsure about the exact cause of this).

You should follow this ticket on GitHub for progress regarding this issue.

Thank you!
Vivek

On Jul 15, 2014, at 12:47 PM, Joel Richardson <[hidden email]> wrote:


Thanks Vivek. I don't know if this applies in our situation because we have tomcat and postgres running of the same server. But maybe the issue creeps in some other way I'm not aware of. Anyway, you've given us something to look into. Thanks again.

Joel
-- 
Joel E. Richardson, Ph.D.
Sr. Research Scientist
Mouse Genome Informatics
The Jackson Laboratory
600 Main Street
Bar Harbor, Maine 04609
207-288-6435

From: <Krishnakumar>, Vivek <[hidden email]>
Date: Tuesday, July 15, 2014 12:21 PM
To: Joel Richardson <[hidden email]>
Cc: "[hidden email]" <[hidden email]>, Intermine Developer List <[hidden email]>, DL AIP <[hidden email]>
Subject: Re: [InterMine Dev] runtime error

Hi Joel,

We actually encountered this exact same issue when we were initially getting ThaleMine (our InterMine interface to Arabidopsis thaliana genome data, https://apps.araport.org/thalemine/) up and running on our production systems.
Similar to your observation, we also did notice that this issue resolved itself temporarily after a Tomcat restart.

After spending a lot of time troubleshooting the issue with the help of the InterMine developers and our in-house IT staff, we arrived at the following conclusion:
We believe the cause of the error (red rectangles of death or RROD) was JCVI software that closed connections between servers sitting on different subnets.
We were able to demonstrate days of uptime without any RROD, merely by using a web server and database server that were on the same subnet (though there were other variables that still need to be ruled out).
We still have the outstanding issue that InterMine software, in particular its connection pool, never recovers from a lost connection.

Hope this provides you with a fresh perspective into this particular issue.

Thank you!

Best Regards,
Vivek

On Behalf of the Araport Team

--
Vivek Krishnakumar
Sr. Bioinformatics Engineer
Plant Genomics

J. Craig Venter Institute
Phone: 301-795-7363
Fax: 301-795-7070

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: runtime error

dmgood11
Just to affirm, yes, the Phytozome team at JGI is also testing out the Hikari connection pool library, now that we’ve been able to secure a setup with pgSQL 9.x

-David


On Tue, Jul 15, 2014 at 10:15 AM, Krishnakumar, Vivek <[hidden email]> wrote:
Hi Joel,

One more thing to consider is that there is ongoing work by the InterMine team to replace the existing JDBC library with a new one, called Hikari Connection Pool library (HikariCP), which should mitigate such issues of dropped/stale connections in the future.
Here is the related GitHub issue: https://github.com/intermine/intermine/issues/558
and the relevant pull request: https://github.com/intermine/intermine/pull/699
All these changes should be in the “beta” branch of the upstream/intermine repository soon.

And there are teams at Cambridge and specific members in our team at JCVI, who are independently testing out this new connection pool library (the Phytozome group at JGI have showed interest in this and are probably testing things out at their end as well).
From some (very) preliminary testing at our end, everything seems to work fine but because of the new library, there seems to be a drastic increase (almost three-fold) in load times (we are unsure about the exact cause of this).

You should follow this ticket on GitHub for progress regarding this issue.

Thank you!
Vivek

On Jul 15, 2014, at 12:47 PM, Joel Richardson <[hidden email]> wrote:


Thanks Vivek. I don't know if this applies in our situation because we have tomcat and postgres running of the same server. But maybe the issue creeps in some other way I'm not aware of. Anyway, you've given us something to look into. Thanks again.

Joel
-- 
Joel E. Richardson, Ph.D.
Sr. Research Scientist
Mouse Genome Informatics
The Jackson Laboratory
600 Main Street
Bar Harbor, Maine 04609
<a href="tel:207-288-6435" value="+12072886435" target="_blank">207-288-6435

From: <Krishnakumar>, Vivek <[hidden email]>
Date: Tuesday, July 15, 2014 12:21 PM
To: Joel Richardson <[hidden email]>
Cc: "[hidden email]" <[hidden email]>, Intermine Developer List <[hidden email]>, DL AIP <[hidden email]>
Subject: Re: [InterMine Dev] runtime error

Hi Joel,

We actually encountered this exact same issue when we were initially getting ThaleMine (our InterMine interface to Arabidopsis thaliana genome data, https://apps.araport.org/thalemine/) up and running on our production systems.
Similar to your observation, we also did notice that this issue resolved itself temporarily after a Tomcat restart.

After spending a lot of time troubleshooting the issue with the help of the InterMine developers and our in-house IT staff, we arrived at the following conclusion:
We believe the cause of the error (red rectangles of death or RROD) was JCVI software that closed connections between servers sitting on different subnets.
We were able to demonstrate days of uptime without any RROD, merely by using a web server and database server that were on the same subnet (though there were other variables that still need to be ruled out).
We still have the outstanding issue that InterMine software, in particular its connection pool, never recovers from a lost connection.

Hope this provides you with a fresh perspective into this particular issue.

Thank you!

Best Regards,
Vivek

On Behalf of the Araport Team

--
Vivek Krishnakumar
Sr. Bioinformatics Engineer
Plant Genomics

J. Craig Venter Institute
Phone: <a href="tel:301-795-7363" value="+13017957363" target="_blank">301-795-7363
Fax: <a href="tel:301-795-7070" value="+13017957070" target="_blank">301-795-7070

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev




--
David M. Goodstein, Ph.D.
Phytozome Group Lead
Plant and Computational Genomics Group
Joint Genome Institute - U.S. Dept. of Energy
Center for Integrative Genomics - UC Berkeley


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: runtime error

Julie Sullivan
In reply to this post by Joel Richardson-2
I've made a ticket:

https://github.com/intermine/intermine/issues/727

But just glancing at the code, all the methods accessing the userprofile
in that class are synchronized except the one mentioned in your
stacktrace. Indeed a duplicate of that method in ProfileManager is
synchronised as well.

> Hi devs,
>
> From time to time, MouseMine gets into a state where users are unable to
> create lists or log in. The error log says:
>
> java.lang.RuntimeException: Unable to load user profileat
> org.intermine.api.profile.TagManager.getUserProfile(TagManager.java:626)at
> org.intermine.api.profile.TagManager.checkUserExists(TagManager.java:93)at
> org.intermine.api.profile.TagManager.getTags(TagManager.java:386)at
> org.intermine.api.template.TemplateManager.getTemplatesWithTag(TemplateManager.java:381)at
> org.intermine.api.template.TemplateManager.getTemplatesWithTag(TemplateManager.java:363)at
> org.intermine.api.template.TemplateManager.getConversionTemplates(TemplateManager.java:347)at
> org.intermine.web.struts.ConvertBagController.execute(ConvertBagController.java:65)at
> etc...
>
> Restarting tomcat seems to clear the problem.
> Any ideas what might be going on?
>
> Thanks,
> Joel
>
> --
> Joel E. Richardson, Ph.D.
> Sr. Research Scientist
> Mouse Genome Informatics
> The Jackson Laboratory
> 600 Main Street
> Bar Harbor, Maine 04609
> 207-288-6435
> [hidden email]
>
> The information in this email, including attachments, may be confidential
> and is intended solely for the addressee(s). If you believe you received
> this email by mistake, please notify the sender by return email as soon as
> possible.
> _______________________________________________
> dev mailing list
> [hidden email]
> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev