too many clients already

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

too many clients already

Pengcheng Yang
Hi miners,

The similar error has been reported previously:
http://gmod.827538.n3.nabble.com/ratmine-data-loading-errors-td4029257.html

However, even I using  project_build script. The error remain.

1) I have set the "db.production.datasource.maxConnections=150" in the
default.intermine.integrate.properties file. During the running of the
project_build, "ps -ef |grep postgres | grep idle |wc -l" return as many
as 167 processes. When I load the source one-by-one, although one source
was failed loaded during to "too many clients" error, the post-process
and the web deployment were successful.

2) I have also tested set the
"db.production.datasource.maxConnections=200". But the error message
suggest this number should not be as larger as 200.

Further I found that many of the process of postgres were "idle". Is it
mean that these processes could be killed? But, I have tested to kill
the idle process, which resulted in the termination of the building process.

3) I have tested to restart postgres server and then run project_build
again. Having tested many times, each time was stopped at different
steps with the same error "too many clients already"

=================== error message============================

BUILD FAILED
/home/pengchy/soft/intermine/imbuild/integrate.xml:54: The following
error occurred while executing this line:
/home/pengchy/soft/intermine/imbuild/source.xml:201: Exception in
XmlDataLoaderTask
         at
org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:170)
         at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
         at org.apache.tools.ant.Task.perform(Task.java:348)
         at org.apache.tools.ant.Target.execute(Target.java:437)
         at org.apache.tools.ant.Target.performTasks(Target.java:458)
         at
org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
         at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:36)
         at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
         at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
         at org.intermine.task.Integrate.performAction(Integrate.java:219)
         at org.intermine.task.Integrate.performAction(Integrate.java:131)
         at org.intermine.task.Integrate.execute(Integrate.java:123)
         at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
         at org.apache.tools.ant.Task.perform(Task.java:348)
         at org.apache.tools.ant.Target.execute(Target.java:437)
         at org.apache.tools.ant.Target.performTasks(Target.java:458)
         at
org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
         at org.apache.tools.ant.Project.executeTarget(Project.java:1376)
         at
org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
         at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
         at org.apache.tools.ant.Main.runBuild(Main.java:857)
         at org.apache.tools.ant.Main.startAnt(Main.java:236)
         at org.apache.tools.ant.launch.Launcher.run(Launcher.java:286)
         at org.apache.tools.ant.launch.Launcher.main(Launcher.java:112)
Caused by: org.intermine.objectstore.ObjectStoreException:
org.intermine.objectstore.ObjectStoreException: Unable to get database
for InterMine ObjectStore
         at
org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:71)
         at
org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:120)
         at
org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:80)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at
org.intermine.dataloader.IntegrationWriterFactory.getIntegrationWriter(IntegrationWriterFactory.java:65)
         at
org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:123)
         ... 31 more
Caused by: org.intermine.objectstore.ObjectStoreException: Unable to get
database for InterMine ObjectStore
         at
org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:358)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at
org.intermine.objectstore.ObjectStoreFactory.getObjectStore(ObjectStoreFactory.java:63)
         at
org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:69)
         ... 39 more
Caused by: java.lang.RuntimeException: Failed to initialise db.production
         at
org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:58)
         at
org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:356)
         ... 45 more
Caused by: com.zaxxer.hikari.pool.PoolInitializationException: Exception
during pool initialization
         at
com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:544)
         at
com.zaxxer.hikari.pool.BaseHikariPool.<init>(BaseHikariPool.java:171)
         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:60)
         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:48)
         at
com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:80)
         at org.intermine.sql.Database.<init>(Database.java:122)
         at
org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:56)
         ... 46 more
Caused by: org.postgresql.util.PSQLException: FATAL: sorry, too many
clients already
         at org.postgresql.Driver$ConnectThread.getResult(Driver.java:355)
         at org.postgresql.Driver.connect(Driver.java:279)
         at java.sql.DriverManager.getConnection(DriverManager.java:664)
         at java.sql.DriverManager.getConnection(DriverManager.java:247)
         at
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
         at
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
         at
com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:438)
         at
com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:542)
         ... 52 more

Total time: 22 seconds
Fri Apr  6 09:55:48 CST 2018

finished


ERROR: 256
failed with exit code 0: ant -v -Dsource=entrez-organism


Best,

Pengcheng


_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: too many clients already

Pengcheng Yang
Hi,

I have resolved this problem by set "max_connections = 500" in
/var/lib/pgsql/data/postgresql.conf file.

Some useful links are:

[1]
https://stackoverflow.com/questions/2757549/org-postgresql-util-psqlexception-fatal-sorry-too-many-clients-already

[2]
https://stackoverflow.com/questions/30778015/how-to-increase-the-max-connections-in-postgres

Best,

Pengcheng


On 2018-4-6 10:20, Pengcheng Yang wrote:

> Hi miners,
>
> The similar error has been reported previously:
> http://gmod.827538.n3.nabble.com/ratmine-data-loading-errors-td4029257.html
>
> However, even I using  project_build script. The error remain.
>
> 1) I have set the "db.production.datasource.maxConnections=150" in the
> default.intermine.integrate.properties file. During the running of the
> project_build, "ps -ef |grep postgres | grep idle |wc -l" return as
> many as 167 processes. When I load the source one-by-one, although one
> source was failed loaded during to "too many clients" error, the
> post-process and the web deployment were successful.
>
> 2) I have also tested set the
> "db.production.datasource.maxConnections=200". But the error message
> suggest this number should not be as larger as 200.
>
> Further I found that many of the process of postgres were "idle". Is
> it mean that these processes could be killed? But, I have tested to
> kill the idle process, which resulted in the termination of the
> building process.
>
> 3) I have tested to restart postgres server and then run project_build
> again. Having tested many times, each time was stopped at different
> steps with the same error "too many clients already"
>
> =================== error message============================
>
> BUILD FAILED
> /home/pengchy/soft/intermine/imbuild/integrate.xml:54: The following
> error occurred while executing this line:
> /home/pengchy/soft/intermine/imbuild/source.xml:201: Exception in
> XmlDataLoaderTask
>         at
> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:170)
>         at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>         at org.apache.tools.ant.Task.perform(Task.java:348)
>         at org.apache.tools.ant.Target.execute(Target.java:437)
>         at org.apache.tools.ant.Target.performTasks(Target.java:458)
>         at
> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>         at
> org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:36)
>         at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>         at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
>         at org.intermine.task.Integrate.performAction(Integrate.java:219)
>         at org.intermine.task.Integrate.performAction(Integrate.java:131)
>         at org.intermine.task.Integrate.execute(Integrate.java:123)
>         at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>         at org.apache.tools.ant.Task.perform(Task.java:348)
>         at org.apache.tools.ant.Target.execute(Target.java:437)
>         at org.apache.tools.ant.Target.performTasks(Target.java:458)
>         at
> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>         at org.apache.tools.ant.Project.executeTarget(Project.java:1376)
>         at
> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>         at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>         at org.apache.tools.ant.Main.runBuild(Main.java:857)
>         at org.apache.tools.ant.Main.startAnt(Main.java:236)
>         at org.apache.tools.ant.launch.Launcher.run(Launcher.java:286)
>         at org.apache.tools.ant.launch.Launcher.main(Launcher.java:112)
> Caused by: org.intermine.objectstore.ObjectStoreException:
> org.intermine.objectstore.ObjectStoreException: Unable to get database
> for InterMine ObjectStore
>         at
> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:71)
>         at
> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:120)
>         at
> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:80)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.intermine.dataloader.IntegrationWriterFactory.getIntegrationWriter(IntegrationWriterFactory.java:65)
>         at
> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:123)
>         ... 31 more
> Caused by: org.intermine.objectstore.ObjectStoreException: Unable to
> get database for InterMine ObjectStore
>         at
> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:358)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.intermine.objectstore.ObjectStoreFactory.getObjectStore(ObjectStoreFactory.java:63)
>         at
> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:69)
>         ... 39 more
> Caused by: java.lang.RuntimeException: Failed to initialise db.production
>         at
> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:58)
>         at
> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:356)
>         ... 45 more
> Caused by: com.zaxxer.hikari.pool.PoolInitializationException:
> Exception during pool initialization
>         at
> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:544)
>         at
> com.zaxxer.hikari.pool.BaseHikariPool.<init>(BaseHikariPool.java:171)
>         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:60)
>         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:48)
>         at
> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:80)
>         at org.intermine.sql.Database.<init>(Database.java:122)
>         at
> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:56)
>         ... 46 more
> Caused by: org.postgresql.util.PSQLException: FATAL: sorry, too many
> clients already
>         at org.postgresql.Driver$ConnectThread.getResult(Driver.java:355)
>         at org.postgresql.Driver.connect(Driver.java:279)
>         at java.sql.DriverManager.getConnection(DriverManager.java:664)
>         at java.sql.DriverManager.getConnection(DriverManager.java:247)
>         at
> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
>         at
> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
>         at
> com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:438)
>         at
> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:542)
>         ... 52 more
>
> Total time: 22 seconds
> Fri Apr  6 09:55:48 CST 2018
>
> finished
>
>
> ERROR: 256
> failed with exit code 0: ant -v -Dsource=entrez-organism
>
>
> Best,
>
> Pengcheng
>
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: too many clients already

Julie Sullivan-2
You shouldn't need 500 connections. Human and flymine build fine with 20.

1. Is your code on github? Can I take a look? Specifically I am
interested in what your data sources are doing.
2. Which version of postgres are you on? I recommend 9.3 but I am using
9.5 without any issue.
3. When this happens, when you see this error, what is the output of `ps
aux | grep postgres`.
4. Do you see any hanging queries? `select * from pg_stat_activity` will
tell you which queries are running.

No you shouldn't need to kill the postgres processes. I would have
thought that your earlier errors might have caused the database
connections to not close properly. This would account for the error
message you saw. However this would have been fixed by a postgres
restart - which you did.

On 04/06/2018 08:48 AM, Pengcheng Yang wrote:

> Hi,
>
> I have resolved this problem by set "max_connections = 500" in
> /var/lib/pgsql/data/postgresql.conf file.
>
> Some useful links are:
>
> [1]
> https://stackoverflow.com/questions/2757549/org-postgresql-util-psqlexception-fatal-sorry-too-many-clients-already 
>
>
> [2]
> https://stackoverflow.com/questions/30778015/how-to-increase-the-max-connections-in-postgres 
>
>
> Best,
>
> Pengcheng
>
>
> On 2018-4-6 10:20, Pengcheng Yang wrote:
>> Hi miners,
>>
>> The similar error has been reported previously:
>> http://gmod.827538.n3.nabble.com/ratmine-data-loading-errors-td4029257.html 
>>
>>
>> However, even I using  project_build script. The error remain.
>>
>> 1) I have set the "db.production.datasource.maxConnections=150" in the
>> default.intermine.integrate.properties file. During the running of the
>> project_build, "ps -ef |grep postgres | grep idle |wc -l" return as
>> many as 167 processes. When I load the source one-by-one, although one
>> source was failed loaded during to "too many clients" error, the
>> post-process and the web deployment were successful.
>>
>> 2) I have also tested set the
>> "db.production.datasource.maxConnections=200". But the error message
>> suggest this number should not be as larger as 200.
>>
>> Further I found that many of the process of postgres were "idle". Is
>> it mean that these processes could be killed? But, I have tested to
>> kill the idle process, which resulted in the termination of the
>> building process.
>>
>> 3) I have tested to restart postgres server and then run project_build
>> again. Having tested many times, each time was stopped at different
>> steps with the same error "too many clients already"
>>
>> =================== error message============================
>>
>> BUILD FAILED
>> /home/pengchy/soft/intermine/imbuild/integrate.xml:54: The following
>> error occurred while executing this line:
>> /home/pengchy/soft/intermine/imbuild/source.xml:201: Exception in
>> XmlDataLoaderTask
>>         at
>> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:170)
>>
>>         at
>> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at
>> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>>
>>         at org.apache.tools.ant.Task.perform(Task.java:348)
>>         at org.apache.tools.ant.Target.execute(Target.java:437)
>>         at org.apache.tools.ant.Target.performTasks(Target.java:458)
>>         at
>> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>>         at
>> org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:36)
>>
>>         at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>>         at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
>>         at org.intermine.task.Integrate.performAction(Integrate.java:219)
>>         at org.intermine.task.Integrate.performAction(Integrate.java:131)
>>         at org.intermine.task.Integrate.execute(Integrate.java:123)
>>         at
>> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at
>> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>>
>>         at org.apache.tools.ant.Task.perform(Task.java:348)
>>         at org.apache.tools.ant.Target.execute(Target.java:437)
>>         at org.apache.tools.ant.Target.performTasks(Target.java:458)
>>         at
>> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>>         at org.apache.tools.ant.Project.executeTarget(Project.java:1376)
>>         at
>> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>>
>>         at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>>         at org.apache.tools.ant.Main.runBuild(Main.java:857)
>>         at org.apache.tools.ant.Main.startAnt(Main.java:236)
>>         at org.apache.tools.ant.launch.Launcher.run(Launcher.java:286)
>>         at org.apache.tools.ant.launch.Launcher.main(Launcher.java:112)
>> Caused by: org.intermine.objectstore.ObjectStoreException:
>> org.intermine.objectstore.ObjectStoreException: Unable to get database
>> for InterMine ObjectStore
>>         at
>> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:71)
>>
>>         at
>> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:120)
>>
>>         at
>> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:80)
>>
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at
>> org.intermine.dataloader.IntegrationWriterFactory.getIntegrationWriter(IntegrationWriterFactory.java:65)
>>
>>         at
>> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:123)
>>
>>         ... 31 more
>> Caused by: org.intermine.objectstore.ObjectStoreException: Unable to
>> get database for InterMine ObjectStore
>>         at
>> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:358)
>>
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at
>> org.intermine.objectstore.ObjectStoreFactory.getObjectStore(ObjectStoreFactory.java:63)
>>
>>         at
>> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:69)
>>
>>         ... 39 more
>> Caused by: java.lang.RuntimeException: Failed to initialise db.production
>>         at
>> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:58)
>>         at
>> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:356)
>>
>>         ... 45 more
>> Caused by: com.zaxxer.hikari.pool.PoolInitializationException:
>> Exception during pool initialization
>>         at
>> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:544)
>>
>>         at
>> com.zaxxer.hikari.pool.BaseHikariPool.<init>(BaseHikariPool.java:171)
>>         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:60)
>>         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:48)
>>         at
>> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:80)
>>         at org.intermine.sql.Database.<init>(Database.java:122)
>>         at
>> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:56)
>>         ... 46 more
>> Caused by: org.postgresql.util.PSQLException: FATAL: sorry, too many
>> clients already
>>         at org.postgresql.Driver$ConnectThread.getResult(Driver.java:355)
>>         at org.postgresql.Driver.connect(Driver.java:279)
>>         at java.sql.DriverManager.getConnection(DriverManager.java:664)
>>         at java.sql.DriverManager.getConnection(DriverManager.java:247)
>>         at
>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
>>
>>         at
>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
>>
>>         at
>> com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:438)
>>
>>         at
>> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:542)
>>
>>         ... 52 more
>>
>> Total time: 22 seconds
>> Fri Apr  6 09:55:48 CST 2018
>>
>> finished
>>
>>
>> ERROR: 256
>> failed with exit code 0: ant -v -Dsource=entrez-organism
>>
>>
>> Best,
>>
>> Pengcheng
>>
>>
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: too many clients already

Pengcheng Yang
Hi Julie,

Thank you for the helpful suggestions.

1. I have just uploaded my source code to github at:
https://github.com/pengchy/intermine/tree/master/testmine for your checking.

2. I deploy the intermine on CentOS 7, PostgreSQL 9.2.3.

3. When this happens, the `ps aux|grep postgres` output the following
message with about >100 postgres process with idle state.

postgres  2035  0.0  0.0 3561652 8792 ?        Ss   15:36   0:00
postgres: postgres userprofile-testmodel 10.4.0.17(47445) idle
postgres  2038  0.0  0.0 3561772 8312 ?        Ss   15:36   0:00
postgres: postgres userprofile-testmodel 10.4.0.17(47446) idle
postgres  2508  0.0  0.0 3561484 8052 ?        Ss   15:37   0:00
postgres: postgres testmine 10.4.0.17(47551) idle
postgres  2579  0.0  0.0 3561640 8532 ?        Ss   15:37   0:00
postgres: postgres userprofile-testmine 10.4.0.17(47584) idle
postgres  2585  0.0  0.0 3561764 8568 ?        Ss   15:37   0:00
postgres: postgres userprofile-testmine 10.4.0.17(47587) idle
postgres 13382  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48353) idle
postgres 13384  0.0  0.0 3560392 4476 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48354) idle
postgres 13386  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48355) idle
postgres 13388  0.0  0.0 3560392 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48356) idle
postgres 13390  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48357) idle
postgres 13392  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48358) idle
postgres 13394  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48359) idle
postgres 13396  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48360) idle
postgres 13398  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48361) idle
postgres 13400  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48362) idle
postgres 13402  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48363) idle
postgres 13404  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48364) idle
postgres 13406  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48365) idle
postgres 13408  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48366) idle
postgres 13410  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48367) idle
postgres 13412  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48368) idle
postgres 13414  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48369) idle
postgres 13416  0.0  0.0 3560392 4476 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48370) idle
postgres 13418  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48371) idle
postgres 13420  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48372) idle
postgres 13422  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48373) idle
postgres 13424  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48374) idle
postgres 13426  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48375) idle
postgres 13428  0.0  0.0 3560392 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48376) idle
postgres 13430  0.0  0.0 3560396 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48377) idle
postgres 13432  0.0  0.0 3560396 4476 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48378) idle
postgres 13434  0.0  0.0 3560396 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48379) idle
postgres 13436  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48380) idle
postgres 13438  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48381) idle
postgres 13440  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48382) idle
postgres 13442  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48383) idle
postgres 13444  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48384) idle
postgres 13446  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48385) idle
postgres 13448  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48386) idle
postgres 13450  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48387) idle
postgres 13452  0.0  0.0 3560396 4472 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48388) idle
postgres 13456  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48389) idle
postgres 13458  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48390) idle
postgres 13460  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48391) idle
postgres 13462  0.0  0.0 3560396 4460 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48392) idle
postgres 13464  0.0  0.0 3560396 4460 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48393) idle
postgres 13466  0.0  0.0 3560396 4460 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48394) idle
postgres 13468  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48395) idle
postgres 13470  0.0  0.0 3560396 4460 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48396) idle
postgres 13472  0.0  0.0 3560396 4460 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48397) idle
postgres 13474  0.0  0.0 3560396 4460 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48398) idle
postgres 13476  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48399) idle
postgres 13478  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48400) idle
postgres 13480  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48401) idle
postgres 13482  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48402) idle
postgres 13484  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48403) idle
postgres 13486  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48404) idle
postgres 13507  0.0  0.0 3560396 4468 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48405) idle
postgres 13509  0.0  0.0 3560396 4476 ?        Ss   17:33   0:00
postgres: postgres testmine 10.4.0.17(48406) idle
postgres 13513  0.0  0.0 3560396 4464 ?        Ss   17:33   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48407) idle
postgres 13619  0.0  0.0 3560396 4472 ?        Ss   17:35   0:00
postgres: postgres testmine 10.4.0.17(48410) idle
postgres 13722  0.0  0.0 3560396 4472 ?        Ss   17:38   0:00
postgres: postgres userprofile-testmine 10.4.0.17(48411) idle
postgres 13848  0.0  0.0 3560396 4468 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48412) idle
postgres 13850  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48413) idle
postgres 13852  0.0  0.0 3560396 4468 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48414) idle
postgres 13854  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48415) idle
postgres 13856  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48416) idle
postgres 13858  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48417) idle
postgres 13860  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48418) idle
postgres 13862  0.0  0.0 3560396 4476 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48419) idle
postgres 13864  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48420) idle
postgres 13866  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres unittest 10.4.0.17(48421) idle
postgres 13870  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres userprofile-testmodel 10.4.0.17(48422) idle
postgres 13872  0.0  0.0 3560396 4476 ?        Ss   17:42   0:00
postgres: postgres userprofile-testmodel 10.4.0.17(48423) idle
postgres 13874  0.0  0.0 3560396 4472 ?        Ss   17:42   0:00
postgres: postgres userprofile-testmodel 10.4.0.17(48424) idle
root     13877  0.0  0.0 112664   980 pts/2    R+   17:42   0:00 grep
--color=auto postgres
postgres 20268  0.0  0.6 3557504 221940 ?      S    15:26   0:04
/usr/bin/postgres -D /var/lib/pgsql/data -p 5432
postgres 20269  0.0  0.0 190472  1520 ?        Ss   15:26   0:00
postgres: logger process
postgres 20271  0.0  2.3 3559456 775564 ?      Ss   15:26   0:01
postgres: checkpointer process
postgres 20272  0.0  0.0 3559008 21280 ?       Ss   15:26   0:00
postgres: writer process
postgres 20273  0.0  0.0 3559008 18296 ?       Ss   15:26   0:01
postgres: wal writer process
postgres 20274  0.0  0.0 3560536 3680 ?        Ss   15:26   0:00
postgres: autovacuum launcher process
postgres 20275  0.0  0.0 193300  2160 ?        Ss   15:26   0:02
postgres: stats collector process

4. After successfully building the InterMine, now, `select * from
pg_stat_activity` returns 76 records as follows.


postgres=# select * from pg_stat_activity;
  datid |        datname        |  pid  | usesysid | usename  |
application_name | client_addr | client_hostname | client_port |        
backend_start
        |          xact_start           | query_start          |        
state_change          | waiting | state  |



query



-------+-----------------------+-------+----------+----------+------------------+-------------+-----------------+-------------+------------------------
-------+-------------------------------+-------------------------------+-------------------------------+---------+--------+----------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
  18438 | testmine              |  2508 |       10 | postgres
|                  | 10.4.0.17   |                 |       47551 |
2018-04-06 15:37:45.155
604+08 |                               | 2018-04-06 15:37:45.25924+08  |
2018-04-06 15:37:45.260696+08 | f       | idle   | SELECT NULL AS
TABLE_CAT, n
.nspname AS TABLE_SCHEM, c.relname AS TABLE_NAME,  CASE n.nspname ~
'^pg_' OR n.nspname = 'information_schema'  WHEN true THEN CASE WHEN
n.nspname = '
pg_catalog' OR n.nspname = 'information_schema' THEN CASE c.relkind  
WHEN 'r' THEN 'SYSTEM TABLE'   WHEN 'v' THEN 'SYSTEM VIEW'   WHEN 'i'
THEN 'SYSTE
M INDEX'   ELSE NULL   END  WHEN n.nspname = 'pg_toast' THEN CASE
c.relkind   WHEN 'r' THEN 'SYSTEM TOAST TABLE'   WHEN 'i' THEN 'SYSTEM
TOAST INDEX'
  ELSE NULL   END  ELSE CASE c.relkind   WHEN 'r' THEN 'TEMPORARY
TABLE'   WHEN 'i' THEN 'TEMPORARY INDEX'   WHEN 'S' THEN 'TEMPORARY
SEQUENCE'   WHEN '
v' THEN 'TEMPORARY VIEW'   ELSE NULL   END  END  WHEN false THEN CASE
c.relkind  WHEN 'r' THEN 'TABLE'  WHEN 'i' THEN 'INDEX'  WHEN 'S' THEN
'SEQUENCE'
   WHEN 'v' THEN 'VIEW'  WHEN 'c' THEN 'TYPE'  WHEN 'f' THEN 'FOREIGN
TABLE'  WHEN 'm' THEN 'MATERIALIZED VIEW'  ELSE NULL END  ELSE NULL 
END  AS TABL
E_TYPE, d.description AS REMARKS  FROM pg_catalog.pg_namespace n,
pg_catalog.pg_class c  L
  18438 | testmine              | 14117 |       10 | postgres
|                  | 10.4.0.17   |                 |       48477 |
2018-04-06 17:44:15.139
331+08 |                               | 2018-04-06 17:44:15.14372+08  |
2018-04-06 17:44:15.143772+08 | f       | idle   | SHOW TRANSACTION
ISOLATION
LEVEL
  18438 | testmine              | 14119 |       10 | postgres
|                  | 10.4.0.17   |                 |       48478 |
2018-04-06 17:44:15.145
721+08 |                               | 2018-04-06 17:44:15.149696+08 |
2018-04-06 17:44:15.149727+08 | f       | idle   | SHOW TRANSACTION
ISOLATION
LEVEL
  18438 | testmine              | 13619 |       10 | postgres
|                  | 10.4.0.17   |                 |       48410 |
2018-04-06 17:35:45.137
484+08 |                               | 2018-04-06 17:35:45.143109+08 |
2018-04-06 17:35:45.143147+08 | f       | idle   | SHOW TRANSACTION
ISOLATION
LEVEL
  18438 | testmine              | 13942 |       10 | postgres
|                  | 10.4.0.17   |                 |       48425 |
2018-04-06 17:43:45.141
417+08 |                               | 2018-04-06 17:43:45.145461+08 |
2018-04-06 17:43:45.145496+08 | f       | idle   | SHOW TRANSACTION
ISOLATION
LEVEL
  18438 | testmine              | 13944 |       10 | postgres
|                  | 10.4.0.17   |                 |       48426 |
2018-04-06 17:43:45.146
769+08 |                               | 2018-04-06 17:43:45.150107+08 |
2018-04-06 17:43:45.150136+08 | f       | idle   | SHOW TRANSACTION
ISOLATION
LEVEL
  18438 | testmine              | 13946 |       10 | postgres
|                  | 10.4.0.17   |                 |       48427 |
2018-04-06 17:43:45.152
21+08  |                               | 2018-04-06 17:43:45.155535+08 |
2018-04-06 17:43:45.155564+08 | f       | idle   | SHOW TRANSACTION
ISOLATION
LEVEL


Best,

Pengcheng


On 2018-4-6 15:50, Julie Sullivan wrote:

> You shouldn't need 500 connections. Human and flymine build fine with 20.
>
> 1. Is your code on github? Can I take a look? Specifically I am
> interested in what your data sources are doing.
> 2. Which version of postgres are you on? I recommend 9.3 but I am
> using 9.5 without any issue.
> 3. When this happens, when you see this error, what is the output of
> `ps aux | grep postgres`.
> 4. Do you see any hanging queries? `select * from pg_stat_activity`
> will tell you which queries are running.
>
> No you shouldn't need to kill the postgres processes. I would have
> thought that your earlier errors might have caused the database
> connections to not close properly. This would account for the error
> message you saw. However this would have been fixed by a postgres
> restart - which you did.
>
> On 04/06/2018 08:48 AM, Pengcheng Yang wrote:
>> Hi,
>>
>> I have resolved this problem by set "max_connections = 500" in
>> /var/lib/pgsql/data/postgresql.conf file.
>>
>> Some useful links are:
>>
>> [1]
>> https://stackoverflow.com/questions/2757549/org-postgresql-util-psqlexception-fatal-sorry-too-many-clients-already 
>>
>>
>> [2]
>> https://stackoverflow.com/questions/30778015/how-to-increase-the-max-connections-in-postgres 
>>
>>
>> Best,
>>
>> Pengcheng
>>
>>
>> On 2018-4-6 10:20, Pengcheng Yang wrote:
>>> Hi miners,
>>>
>>> The similar error has been reported previously:
>>> http://gmod.827538.n3.nabble.com/ratmine-data-loading-errors-td4029257.html 
>>>
>>>
>>> However, even I using  project_build script. The error remain.
>>>
>>> 1) I have set the "db.production.datasource.maxConnections=150" in
>>> the default.intermine.integrate.properties file. During the running
>>> of the project_build, "ps -ef |grep postgres | grep idle |wc -l"
>>> return as many as 167 processes. When I load the source one-by-one,
>>> although one source was failed loaded during to "too many clients"
>>> error, the post-process and the web deployment were successful.
>>>
>>> 2) I have also tested set the
>>> "db.production.datasource.maxConnections=200". But the error message
>>> suggest this number should not be as larger as 200.
>>>
>>> Further I found that many of the process of postgres were "idle". Is
>>> it mean that these processes could be killed? But, I have tested to
>>> kill the idle process, which resulted in the termination of the
>>> building process.
>>>
>>> 3) I have tested to restart postgres server and then run
>>> project_build again. Having tested many times, each time was stopped
>>> at different steps with the same error "too many clients already"
>>>
>>> =================== error message============================
>>>
>>> BUILD FAILED
>>> /home/pengchy/soft/intermine/imbuild/integrate.xml:54: The following
>>> error occurred while executing this line:
>>> /home/pengchy/soft/intermine/imbuild/source.xml:201: Exception in
>>> XmlDataLoaderTask
>>>         at
>>> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:170)
>>>
>>>         at
>>> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>>>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at
>>> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>>>
>>>         at org.apache.tools.ant.Task.perform(Task.java:348)
>>>         at org.apache.tools.ant.Target.execute(Target.java:437)
>>>         at org.apache.tools.ant.Target.performTasks(Target.java:458)
>>>         at
>>> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>>>         at
>>> org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:36)
>>>
>>>         at
>>> org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>>>         at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
>>>         at
>>> org.intermine.task.Integrate.performAction(Integrate.java:219)
>>>         at
>>> org.intermine.task.Integrate.performAction(Integrate.java:131)
>>>         at org.intermine.task.Integrate.execute(Integrate.java:123)
>>>         at
>>> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>>>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at
>>> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>>>
>>>         at org.apache.tools.ant.Task.perform(Task.java:348)
>>>         at org.apache.tools.ant.Target.execute(Target.java:437)
>>>         at org.apache.tools.ant.Target.performTasks(Target.java:458)
>>>         at
>>> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>>>         at
>>> org.apache.tools.ant.Project.executeTarget(Project.java:1376)
>>>         at
>>> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>>>
>>>         at
>>> org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>>>         at org.apache.tools.ant.Main.runBuild(Main.java:857)
>>>         at org.apache.tools.ant.Main.startAnt(Main.java:236)
>>>         at org.apache.tools.ant.launch.Launcher.run(Launcher.java:286)
>>>         at org.apache.tools.ant.launch.Launcher.main(Launcher.java:112)
>>> Caused by: org.intermine.objectstore.ObjectStoreException:
>>> org.intermine.objectstore.ObjectStoreException: Unable to get
>>> database for InterMine ObjectStore
>>>         at
>>> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:71)
>>>
>>>         at
>>> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:120)
>>>
>>>         at
>>> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:80)
>>>
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at
>>> org.intermine.dataloader.IntegrationWriterFactory.getIntegrationWriter(IntegrationWriterFactory.java:65)
>>>
>>>         at
>>> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:123)
>>>
>>>         ... 31 more
>>> Caused by: org.intermine.objectstore.ObjectStoreException: Unable to
>>> get database for InterMine ObjectStore
>>>         at
>>> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:358)
>>>
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at
>>> org.intermine.objectstore.ObjectStoreFactory.getObjectStore(ObjectStoreFactory.java:63)
>>>
>>>         at
>>> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:69)
>>>
>>>         ... 39 more
>>> Caused by: java.lang.RuntimeException: Failed to initialise
>>> db.production
>>>         at
>>> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:58)
>>>         at
>>> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:356)
>>>
>>>         ... 45 more
>>> Caused by: com.zaxxer.hikari.pool.PoolInitializationException:
>>> Exception during pool initialization
>>>         at
>>> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:544)
>>>
>>>         at
>>> com.zaxxer.hikari.pool.BaseHikariPool.<init>(BaseHikariPool.java:171)
>>>         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:60)
>>>         at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:48)
>>>         at
>>> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:80)
>>>         at org.intermine.sql.Database.<init>(Database.java:122)
>>>         at
>>> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:56)
>>>         ... 46 more
>>> Caused by: org.postgresql.util.PSQLException: FATAL: sorry, too many
>>> clients already
>>>         at
>>> org.postgresql.Driver$ConnectThread.getResult(Driver.java:355)
>>>         at org.postgresql.Driver.connect(Driver.java:279)
>>>         at java.sql.DriverManager.getConnection(DriverManager.java:664)
>>>         at java.sql.DriverManager.getConnection(DriverManager.java:247)
>>>         at
>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
>>>
>>>         at
>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
>>>
>>>         at
>>> com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:438)
>>>
>>>         at
>>> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:542)
>>>
>>>         ... 52 more
>>>
>>> Total time: 22 seconds
>>> Fri Apr  6 09:55:48 CST 2018
>>>
>>> finished
>>>
>>>
>>> ERROR: 256
>>> failed with exit code 0: ant -v -Dsource=entrez-organism
>>>
>>>
>>> Best,
>>>
>>> Pengcheng
>>>
>>>
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: too many clients already

Julie Sullivan-2
In reply to this post by Pengcheng Yang
For the record this was solved by not having a webapp running using the
same database that is being used by the build.

On 04/06/2018 03:20 AM, Pengcheng Yang wrote:

> Hi miners,
>
> The similar error has been reported previously:
> http://gmod.827538.n3.nabble.com/ratmine-data-loading-errors-td4029257.html
>
> However, even I using  project_build script. The error remain.
>
> 1) I have set the "db.production.datasource.maxConnections=150" in the
> default.intermine.integrate.properties file. During the running of the
> project_build, "ps -ef |grep postgres | grep idle |wc -l" return as many
> as 167 processes. When I load the source one-by-one, although one source
> was failed loaded during to "too many clients" error, the post-process
> and the web deployment were successful.
>
> 2) I have also tested set the
> "db.production.datasource.maxConnections=200". But the error message
> suggest this number should not be as larger as 200.
>
> Further I found that many of the process of postgres were "idle". Is it
> mean that these processes could be killed? But, I have tested to kill
> the idle process, which resulted in the termination of the building
> process.
>
> 3) I have tested to restart postgres server and then run project_build
> again. Having tested many times, each time was stopped at different
> steps with the same error "too many clients already"
>
> =================== error message============================
>
> BUILD FAILED
> /home/pengchy/soft/intermine/imbuild/integrate.xml:54: The following
> error occurred while executing this line:
> /home/pengchy/soft/intermine/imbuild/source.xml:201: Exception in
> XmlDataLoaderTask
>          at
> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:170)
>
>          at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>          at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>          at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>          at org.apache.tools.ant.Task.perform(Task.java:348)
>          at org.apache.tools.ant.Target.execute(Target.java:437)
>          at org.apache.tools.ant.Target.performTasks(Target.java:458)
>          at
> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>          at
> org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:36)
>
>          at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>          at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
>          at org.intermine.task.Integrate.performAction(Integrate.java:219)
>          at org.intermine.task.Integrate.performAction(Integrate.java:131)
>          at org.intermine.task.Integrate.execute(Integrate.java:123)
>          at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
>          at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>          at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>          at org.apache.tools.ant.Task.perform(Task.java:348)
>          at org.apache.tools.ant.Target.execute(Target.java:437)
>          at org.apache.tools.ant.Target.performTasks(Target.java:458)
>          at
> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
>          at org.apache.tools.ant.Project.executeTarget(Project.java:1376)
>          at
> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>
>          at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
>          at org.apache.tools.ant.Main.runBuild(Main.java:857)
>          at org.apache.tools.ant.Main.startAnt(Main.java:236)
>          at org.apache.tools.ant.launch.Launcher.run(Launcher.java:286)
>          at org.apache.tools.ant.launch.Launcher.main(Launcher.java:112)
> Caused by: org.intermine.objectstore.ObjectStoreException:
> org.intermine.objectstore.ObjectStoreException: Unable to get database
> for InterMine ObjectStore
>          at
> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:71)
>
>          at
> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:120)
>
>          at
> org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:80)
>
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>          at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at
> org.intermine.dataloader.IntegrationWriterFactory.getIntegrationWriter(IntegrationWriterFactory.java:65)
>
>          at
> org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:123)
>
>          ... 31 more
> Caused by: org.intermine.objectstore.ObjectStoreException: Unable to get
> database for InterMine ObjectStore
>          at
> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:358)
>
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>          at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at
> org.intermine.objectstore.ObjectStoreFactory.getObjectStore(ObjectStoreFactory.java:63)
>
>          at
> org.intermine.objectstore.ObjectStoreWriterFactory.getObjectStoreWriter(ObjectStoreWriterFactory.java:69)
>
>          ... 39 more
> Caused by: java.lang.RuntimeException: Failed to initialise db.production
>          at
> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:58)
>          at
> org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.getInstance(ObjectStoreInterMineImpl.java:356)
>
>          ... 45 more
> Caused by: com.zaxxer.hikari.pool.PoolInitializationException: Exception
> during pool initialization
>          at
> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:544)
>
>          at
> com.zaxxer.hikari.pool.BaseHikariPool.<init>(BaseHikariPool.java:171)
>          at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:60)
>          at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:48)
>          at
> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:80)
>          at org.intermine.sql.Database.<init>(Database.java:122)
>          at
> org.intermine.sql.DatabaseFactory.getDatabase(DatabaseFactory.java:56)
>          ... 46 more
> Caused by: org.postgresql.util.PSQLException: FATAL: sorry, too many
> clients already
>          at org.postgresql.Driver$ConnectThread.getResult(Driver.java:355)
>          at org.postgresql.Driver.connect(Driver.java:279)
>          at java.sql.DriverManager.getConnection(DriverManager.java:664)
>          at java.sql.DriverManager.getConnection(DriverManager.java:247)
>          at
> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
>
>          at
> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
>
>          at
> com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:438)
>
>          at
> com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:542)
>
>          ... 52 more
>
> Total time: 22 seconds
> Fri Apr  6 09:55:48 CST 2018
>
> finished
>
>
> ERROR: 256
> failed with exit code 0: ant -v -Dsource=entrez-organism
>
>
> Best,
>
> Pengcheng
>
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev