Error in worker thread

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Error in worker thread

joe carlson
Hi Folks,

 From time to time I've been seeing an ObjectStoreException with this
message: "Error in worker thread". I'm using a loader that I've
subclassed from DirectDataLoader. I've done enough tracing to see that
it comes from org.intermine.dataloader.ParallelBatchingFetcher,
specifically an exception raised at the jobs.put call in the code block

>         synchronized (pksNotDone) {
>             while (!pksToDo.isEmpty()) {
>                 Iterator<PrimaryKey> pkIter = pksToDo.keySet().iterator();
>                 while (pkIter.hasNext()) {
>                     PrimaryKey pk = pkIter.next();
>                     ClassDescriptor cld = pksToDo.get(pk);
>                     if (canDoPkNow(pk, cld, pksNotDone)) {
>                         jobs.put(new WorkUnit(pk, cld, results,
> cldToObjectsForCld.get(cld),
>                                     pksNotDone, exceptions));
>                         pkIter.remove();
>                     //} else {
>                     //    LOG.error("Cannot do pk " + cld.getName() +
> "." + pk.getName() + " yet");
>                     }
>                 }
>                 try {
>                     pksNotDone.wait();
>                 } catch (InterruptedException e) {
>                 }
>                 if (!exceptions.isEmpty()) {

At the end of this block exceptions is not empty, but in my debug
version getMessage() on the exceptions in the collection (there appear
to be 2) is null.

It's a long running piece of code - about 8 hours before I hit the
exception - and running it through the debugger is painful. I'm about to
start it up to see if I can see more details on the exception to see why
this is thrown.  But while I'm doing so I'm curious if anyone can give
me insight as to why this gets generated if they've seen it before.

Thanks,

Joe
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Error in worker thread

Julie Sullivan-2
Yes, I see this error quite a bit when there is a duplicate primary key:

Caused by: org.intermine.objectstore.ObjectStoreException: Error in
worker thread
          at
org.intermine.dataloader.ParallelBatchingFetcher.doPks(ParallelBatchingFetcher.java:99)
          at
org.intermine.dataloader.BatchingFetcher.getEquivalentsFor(BatchingFetcher.java:240)
          at
org.intermine.dataloader.BatchingFetcher$NoseyObjectStore.execute(BatchingFetcher.java:562)
          at
org.intermine.objectstore.query.ResultsBatches.fetchBatchFromObjectStore(ResultsBatches.java:192)
          at
org.intermine.objectstore.query.PrefetchManager.doRequest(PrefetchManager.java:240)
          at
org.intermine.objectstore.query.PrefetchManager.doRequest(PrefetchManager.java:145)
          at
org.intermine.objectstore.query.ResultsBatches.getBatch(ResultsBatches.java:164)
          at
org.intermine.objectstore.query.ResultsBatches.getRowsFromBatch(ResultsBatches.java:139)
          at org.intermine.objectstore.query.Results.range(Results.java:196)
          at
org.intermine.objectstore.query.SingletonResults.range(SingletonResults.java:65)
          at org.intermine.objectstore.query.Results.get(Results.java:225)
          ... 35 more
Caused by: org.intermine.objectstore.ObjectStoreException: Duplicate
objects found for pk org.intermine.model.bio.Gene.key_primaryidentifier:
Gene [briefDescription="null", chromosome=null, chromosomeLocation=null,
description="null", downstreamIntergenicRegion=null, id="49",
length="null", mgiType="null", name="null", ncbiGeneNumber="null",
organism=39618, primaryIdentifier="C2", score="null", scoreType="null",
secondaryIdentifier="null", sequence=null, sequenceOntologyTerm=39621,
symbol="null", upstreamIntergenicRegion=null]
          at
org.intermine.dataloader.BatchingFetcher.doPk(BatchingFetcher.java:424)
          at
org.intermine.dataloader.ParallelBatchingFetcher$WorkUnit.fetch(ParallelBatchingFetcher.java:142)
          at
org.intermine.dataloader.ParallelBatchingFetcher$Worker.run(ParallelBatchingFetcher.java:163)
          at java.lang.Thread.run(Thread.java:636)

----

Does your stacktrace look different?


On 05/02/2018 05:45 PM, Joe Carlson wrote:

> Hi Folks,
>
>  From time to time I've been seeing an ObjectStoreException with this
> message: "Error in worker thread". I'm using a loader that I've
> subclassed from DirectDataLoader. I've done enough tracing to see that
> it comes from org.intermine.dataloader.ParallelBatchingFetcher,
> specifically an exception raised at the jobs.put call in the code block
>
>>         synchronized (pksNotDone) {
>>             while (!pksToDo.isEmpty()) {
>>                 Iterator<PrimaryKey> pkIter =
>> pksToDo.keySet().iterator();
>>                 while (pkIter.hasNext()) {
>>                     PrimaryKey pk = pkIter.next();
>>                     ClassDescriptor cld = pksToDo.get(pk);
>>                     if (canDoPkNow(pk, cld, pksNotDone)) {
>>                         jobs.put(new WorkUnit(pk, cld, results,
>> cldToObjectsForCld.get(cld),
>>                                     pksNotDone, exceptions));
>>                         pkIter.remove();
>>                     //} else {
>>                     //    LOG.error("Cannot do pk " + cld.getName() +
>> "." + pk.getName() + " yet");
>>                     }
>>                 }
>>                 try {
>>                     pksNotDone.wait();
>>                 } catch (InterruptedException e) {
>>                 }
>>                 if (!exceptions.isEmpty()) {
>
> At the end of this block exceptions is not empty, but in my debug
> version getMessage() on the exceptions in the collection (there appear
> to be 2) is null.
>
> It's a long running piece of code - about 8 hours before I hit the
> exception - and running it through the debugger is painful. I'm about to
> start it up to see if I can see more details on the exception to see why
> this is thrown.  But while I'm doing so I'm curious if anyone can give
> me insight as to why this gets generated if they've seen it before.
>
> Thanks,
>
> Joe
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Error in worker thread

joe carlson
Hi Julie,

Thanks for you info. My stack trace looked different. But unfortunately
I did not save the message. I do recall having a second level of 'Caused
by:' in the output.

I added some code that tried to show more info when shutting down. And,
as luck would have it, everything worked just fine. I had seen this
error a couple times before and every time I tried to debug it by
increasing the verbosity of the output, it worked. It may have been a
problem with the postgres server dropping a connection or some other
error. If I see it again I'll try to save the stack trace.

Thanks,

Joe

On 05/03/2018 01:35 AM, Julie Sullivan wrote:

> Yes, I see this error quite a bit when there is a duplicate primary key:
>
> Caused by: org.intermine.objectstore.ObjectStoreException: Error in
> worker thread
>          at
> org.intermine.dataloader.ParallelBatchingFetcher.doPks(ParallelBatchingFetcher.java:99)
>          at
> org.intermine.dataloader.BatchingFetcher.getEquivalentsFor(BatchingFetcher.java:240)
>          at
> org.intermine.dataloader.BatchingFetcher$NoseyObjectStore.execute(BatchingFetcher.java:562)
>          at
> org.intermine.objectstore.query.ResultsBatches.fetchBatchFromObjectStore(ResultsBatches.java:192)
>          at
> org.intermine.objectstore.query.PrefetchManager.doRequest(PrefetchManager.java:240)
>          at
> org.intermine.objectstore.query.PrefetchManager.doRequest(PrefetchManager.java:145)
>          at
> org.intermine.objectstore.query.ResultsBatches.getBatch(ResultsBatches.java:164)
>          at
> org.intermine.objectstore.query.ResultsBatches.getRowsFromBatch(ResultsBatches.java:139)
>          at
> org.intermine.objectstore.query.Results.range(Results.java:196)
>          at
> org.intermine.objectstore.query.SingletonResults.range(SingletonResults.java:65)
>          at org.intermine.objectstore.query.Results.get(Results.java:225)
>          ... 35 more
> Caused by: org.intermine.objectstore.ObjectStoreException: Duplicate
> objects found for pk
> org.intermine.model.bio.Gene.key_primaryidentifier: Gene
> [briefDescription="null", chromosome=null, chromosomeLocation=null,
> description="null", downstreamIntergenicRegion=null, id="49",
> length="null", mgiType="null", name="null", ncbiGeneNumber="null",
> organism=39618, primaryIdentifier="C2", score="null",
> scoreType="null", secondaryIdentifier="null", sequence=null,
> sequenceOntologyTerm=39621, symbol="null", upstreamIntergenicRegion=null]
>          at
> org.intermine.dataloader.BatchingFetcher.doPk(BatchingFetcher.java:424)
>          at
> org.intermine.dataloader.ParallelBatchingFetcher$WorkUnit.fetch(ParallelBatchingFetcher.java:142)
>          at
> org.intermine.dataloader.ParallelBatchingFetcher$Worker.run(ParallelBatchingFetcher.java:163)
>          at java.lang.Thread.run(Thread.java:636)
>
> ----
>
> Does your stacktrace look different?
>
>
> On 05/02/2018 05:45 PM, Joe Carlson wrote:
>> Hi Folks,
>>
>>  From time to time I've been seeing an ObjectStoreException with this
>> message: "Error in worker thread". I'm using a loader that I've
>> subclassed from DirectDataLoader. I've done enough tracing to see
>> that it comes from org.intermine.dataloader.ParallelBatchingFetcher,
>> specifically an exception raised at the jobs.put call in the code block
>>
>>>         synchronized (pksNotDone) {
>>>             while (!pksToDo.isEmpty()) {
>>>                 Iterator<PrimaryKey> pkIter =
>>> pksToDo.keySet().iterator();
>>>                 while (pkIter.hasNext()) {
>>>                     PrimaryKey pk = pkIter.next();
>>>                     ClassDescriptor cld = pksToDo.get(pk);
>>>                     if (canDoPkNow(pk, cld, pksNotDone)) {
>>>                         jobs.put(new WorkUnit(pk, cld, results,
>>> cldToObjectsForCld.get(cld),
>>>                                     pksNotDone, exceptions));
>>>                         pkIter.remove();
>>>                     //} else {
>>>                     //    LOG.error("Cannot do pk " + cld.getName()
>>> + "." + pk.getName() + " yet");
>>>                     }
>>>                 }
>>>                 try {
>>>                     pksNotDone.wait();
>>>                 } catch (InterruptedException e) {
>>>                 }
>>>                 if (!exceptions.isEmpty()) {
>>
>> At the end of this block exceptions is not empty, but in my debug
>> version getMessage() on the exceptions in the collection (there
>> appear to be 2) is null.
>>
>> It's a long running piece of code - about 8 hours before I hit the
>> exception - and running it through the debugger is painful. I'm about
>> to start it up to see if I can see more details on the exception to
>> see why this is thrown.  But while I'm doing so I'm curious if anyone
>> can give me insight as to why this gets generated if they've seen it
>> before.
>>
>> Thanks,
>>
>> Joe
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev