data loading errors

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

data loading errors

kkarra@stanford.edu
Hello,

If loading a data source that takes a while, I see this type of warning and eventually connection timeout, wondering how to get around this.

[dataLoad] [db.production housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - db.production - Thread starvation or clock leap detected (housekeeper delta=58s402ms).
 [dataLoad] [db.production housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - db.production - Thread starvation or clock leap detected (housekeeper delta=46s330ms).


Connection error:
org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.Driver$ConnectThread.getResult(Driver.java:403)
        at org.postgresql.Driver.connect(Driver.java:261)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
        at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
        at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:706)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:692)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException
        at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.postgresql.core.PGStream.<init>(PGStream.java:69)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.access$100(Driver.java:61)
        at org.postgresql.Driver$ConnectThread.run(Driver.java:362)
        ... 1 more
Failed to create a Non-Pooling DataSource from PostgreSQL JDBC Driver 42.2.2 for postgres at jdbc:postgresql

Thanks,
-Kalpana

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: data loading errors

Sam Hokin-3
I get that when the database/connections are heavily loaded, like if I'm running post-processors and sources at the same time.
Sometimes it recovers with just the WARNs, and sometimes it'll drop the connection with a socket timeout as you see. You'll want to
boost your resources, likely on the database server, but perhaps also on the machine running the IM code.

On 10/26/20 11:28 PM, Kalpana Karra wrote:

> Hello,
>
> If loading a data source that takes a while, I see this type of warning and eventually connection timeout, wondering how to get around this.
>
> [dataLoad] [db.production housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - db.production - Thread starvation or clock leap detected (housekeeper delta=58s402ms).
>   [dataLoad] [db.production housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - db.production - Thread starvation or clock leap detected (housekeeper delta=46s330ms).
>
>
> Connection error:
> org.postgresql.util.PSQLException: The connection attempt failed.
> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:403)
> at org.postgresql.Driver.connect(Driver.java:261)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
> at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
> at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
> at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
> at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
> at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
> at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:706)
> at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:692)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketTimeoutException
> at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:607)
> at org.postgresql.core.PGStream.<init>(PGStream.java:69)
> at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
> at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
> at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
> at org.postgresql.Driver.makeConnection(Driver.java:452)
> at org.postgresql.Driver.access$100(Driver.java:61)
> at org.postgresql.Driver$ConnectThread.run(Driver.java:362)
> ... 1 more
> Failed to create a Non-Pooling DataSource from PostgreSQL JDBC Driver 42.2.2 for postgres at jdbc:postgresql
>
> Thanks,
> -Kalpana
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: data loading errors

kkarra@stanford.edu
Thanks, Sam! trying with increased -Xmx..
Interestingly enough, project_build goes on to load the next data source and data is getting into the database for the source on which this error occurs. It happens on one of the longest data source.



> On Oct 27, 2020, at 8:53 AM, Sam Hokin <[hidden email]> wrote:
>
> I get that when the database/connections are heavily loaded, like if I'm running post-processors and sources at the same time. Sometimes it recovers with just the WARNs, and sometimes it'll drop the connection with a socket timeout as you see. You'll want to boost your resources, likely on the database server, but perhaps also on the machine running the IM code.
>
> On 10/26/20 11:28 PM, Kalpana Karra wrote:
>> Hello,
>> If loading a data source that takes a while, I see this type of warning and eventually connection timeout, wondering how to get around this.
>> [dataLoad] [db.production housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - db.production - Thread starvation or clock leap detected (housekeeper delta=58s402ms).
>>  [dataLoad] [db.production housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - db.production - Thread starvation or clock leap detected (housekeeper delta=46s330ms).
>> Connection error:
>> org.postgresql.util.PSQLException: The connection attempt failed.
>> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:403)
>> at org.postgresql.Driver.connect(Driver.java:261)
>> at java.sql.DriverManager.getConnection(DriverManager.java:664)
>> at java.sql.DriverManager.getConnection(DriverManager.java:247)
>> at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
>> at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
>> at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
>> at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
>> at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
>> at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
>> at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:706)
>> at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:692)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.net.SocketTimeoutException
>> at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:607)
>> at org.postgresql.core.PGStream.<init>(PGStream.java:69)
>> at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
>> at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
>> at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
>> at org.postgresql.Driver.makeConnection(Driver.java:452)
>> at org.postgresql.Driver.access$100(Driver.java:61)
>> at org.postgresql.Driver$ConnectThread.run(Driver.java:362)
>> ... 1 more
>> Failed to create a Non-Pooling DataSource from PostgreSQL JDBC Driver 42.2.2 for postgres at jdbc:postgresql
>> Thanks,
>> -Kalpana
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev