burning through id's

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

burning through id's

joe carlson
Hello,

I'm getting more and more concerned with running out of id’s in the database. To this end I'm presently working on modifications to the codebase to optionally allow for 64-bit ids (I can load and post-process malariamine; there are a few other issues to get the webapp up.) More on that later.

In the meantime, I've noticed another issue that has caused my id counter to go negative. I can run through id’s when the web app is up. I see the message in the log file saying “Got a new set of serial number” from ObjectStoreInterMineImpl in the tomcat log file. What triggers this? Does it happen with every web app restart? Is this used in storing lists/precomputes/query batching?

For our past couple of minor releases, I’ve been adding data to the previous version. (I start by restoring the db dump on a new build system, add data to it and then switch over the public site to that.) Now I see that I have huge gaps in the ids corresponding to the time that the web app was up. Big gaps of 700,000,000.

In retrospect I should have checked the value of the sequence before entering more data into the mine. That would have prevented the holes and the fact that my id’s are now negative. But beside that, it’s not unreasonable to expect that a web app could run through all ids with enough restarts.

Any thoughts on this? What is the serial numbers used for in the web app?

Thanks,

Joe Carlson
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: burning through id's

Justin Clark-Casey-2
Hi Joe.  I don't see this happening on a fresh restart of Synbiomine but I did see it happening at points in the log of a previous longer-running instance.

In my case, the messages about serial number generation are not obviously related to any other log messages (though there aren't many of those - I don't know if
you have more detail with log levels turned up to debug).  I wouldn't expect the webapp to require new serial numbers - no new bioentities or clobs are being
written and bags/precomputes are identified by unique table names using the separate objectstore_unique_integer sequence rather than serial.

But clearly something is happening, perhaps (judging from my log) something periodic that is accidentally requesting IDs but doesn't actually need them.

I did make a small change to the serial number message in github dev just to show which sequence number is being taken.  But to actually pin down what is
happening could you temporarily insert a stack trace log when the new sequence number is being requested?  That would hopefully give us a toehold on the code
path making these requests.

--
Justin Clark-Casey, Synbiomine/InterMine Developer
http://synbiomine.org
http://twitter.com/justincc

On 03/06/16 23:34, Joe Carlson wrote:

> Hello,
>
> I'm getting more and more concerned with running out of id’s in the database. To this end I'm presently working on modifications to the codebase to optionally allow for 64-bit ids (I can load and post-process malariamine; there are a few other issues to get the webapp up.) More on that later.
>
> In the meantime, I've noticed another issue that has caused my id counter to go negative. I can run through id’s when the web app is up. I see the message in the log file saying “Got a new set of serial number” from ObjectStoreInterMineImpl in the tomcat log file. What triggers this? Does it happen with every web app restart? Is this used in storing lists/precomputes/query batching?
>
> For our past couple of minor releases, I’ve been adding data to the previous version. (I start by restoring the db dump on a new build system, add data to it and then switch over the public site to that.) Now I see that I have huge gaps in the ids corresponding to the time that the web app was up. Big gaps of 700,000,000.
>
> In retrospect I should have checked the value of the sequence before entering more data into the mine. That would have prevented the holes and the fact that my id’s are now negative. But beside that, it’s not unreasonable to expect that a web app could run through all ids with enough restarts.
>
> Any thoughts on this? What is the serial numbers used for in the web app?
>
> Thanks,
>
> Joe Carlson
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: burning through id's

joe carlson


On 06/07/2016 10:53 AM, Justin Clark-Casey wrote:

> Hi Joe.  I don't see this happening on a fresh restart of Synbiomine
> but I did see it happening at points in the log of a previous
> longer-running instance.
>
> In my case, the messages about serial number generation are not
> obviously related to any other log messages (though there aren't many
> of those - I don't know if you have more detail with log levels turned
> up to debug).  I wouldn't expect the webapp to require new serial
> numbers - no new bioentities or clobs are being written and
> bags/precomputes are identified by unique table names using the
> separate objectstore_unique_integer sequence rather than serial.
>
> But clearly something is happening, perhaps (judging from my log)
> something periodic that is accidentally requesting IDs but doesn't
> actually need them.
Hi Justin,

I just fired up the web app in the debugger and was looking to see where
I hit a breakpoint on getSerialWithConnection. I did not hit the
breakpoint on startup, but did when I tried to save a list. After
pasting in 10 gene identifiers, I hit it after hitting the 'save a list
of 10 genes'  The stack trace showed that InterMineBag.init() calls
ObjectStore.createObjectStoreBag() and that creates a new
ObjectStoreBag() with an argument getSerial().intValue()

The current state of my mine is that I'm about 3/4 the way through my
id's. Here is the "histogram" of my intermineobject id's:

phytomine-11.0.5=# select 100000000*((id/100000000)::int),count(*) from
intermineobject group by 1 order by 1;
   ?column?   |   count
-------------+-----------
  -2100000000 |  22041400
  -1400000000 |         2
  -1000000000 |   9002637
   -800000000 |  25893789
   -700000000 |   8123453
            0 |  50039666
    100000000 |  63311389
    200000000 |  62498622
    300000000 |  64681926
    400000000 | 100000000
    500000000 |  99093219
    600000000 |  98746936
    700000000 |  98667077
    800000000 | 100000000
    900000000 |  95211450
   1000000000 | 100000000
   1100000000 |  52068134
   1200000000 |  52786551
   1300000000 |  56096351
   1400000000 |  35956705
   2100000000 |    738133
(21 rows)

The big jump from 1.4G to 2.1G was after the second loading process.
Then the id wrapped and went negative. A second and third big jump
happened in the third and fourth loading process. We currently have ~500
saved lists in the user profile database that were created after this
mine build. What I'm seeing in the gaps indicate it should be more like
1200 lists; maybe there were lists created but never saved.

What is flymine like? How many lists are created between builds of the
mine? And do you see issues of the id wrapping around?

By the way, I did need to make a few changes to get the web app to
report objects with negative id's. In
org.intermine.web.struts.ReportController.getRequestedObject, for
example, the call to StringUtils.isNumeric returns false of the argument
is a negative integer.

I'm OK for now. And I think that I'm going to do a full rebuild before
the id counter fully wraps around. It is nonetheless a concern that the
serial is getting incremented and not being used.

Thanks,

Joe

>
> I did make a small change to the serial number message in github dev
> just to show which sequence number is being taken.  But to actually
> pin down what is happening could you temporarily insert a stack trace
> log when the new sequence number is being requested? That would
> hopefully give us a toehold on the code path making these requests.
>
> --
> Justin Clark-Casey, Synbiomine/InterMine Developer
> http://synbiomine.org
> http://twitter.com/justincc
>
> On 03/06/16 23:34, Joe Carlson wrote:
>> Hello,
>>
>> I'm getting more and more concerned with running out of id’s in the
>> database. To this end I'm presently working on modifications to the
>> codebase to optionally allow for 64-bit ids (I can load and
>> post-process malariamine; there are a few other issues to get the
>> webapp up.) More on that later.
>>
>> In the meantime, I've noticed another issue that has caused my id
>> counter to go negative. I can run through id’s when the web app is
>> up. I see the message in the log file saying “Got a new set of serial
>> number” from ObjectStoreInterMineImpl in the tomcat log file. What
>> triggers this? Does it happen with every web app restart? Is this
>> used in storing lists/precomputes/query batching?
>>
>> For our past couple of minor releases, I’ve been adding data to the
>> previous version. (I start by restoring the db dump on a new build
>> system, add data to it and then switch over the public site to that.)
>> Now I see that I have huge gaps in the ids corresponding to the time
>> that the web app was up. Big gaps of 700,000,000.
>>
>> In retrospect I should have checked the value of the sequence before
>> entering more data into the mine. That would have prevented the holes
>> and the fact that my id’s are now negative. But beside that, it’s not
>> unreasonable to expect that a web app could run through all ids with
>> enough restarts.
>>
>> Any thoughts on this? What is the serial numbers used for in the web
>> app?
>>
>> Thanks,
>>
>> Joe Carlson
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
>>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: burning through id's

Justin Clark-Casey-2
On 13/06/16 20:13, Joe Carlson wrote:

>
>
> On 06/07/2016 10:53 AM, Justin Clark-Casey wrote:
>> Hi Joe.  I don't see this happening on a fresh restart of Synbiomine but I did see it happening at points in the log of a previous longer-running instance.
>>
>> In my case, the messages about serial number generation are not obviously related to any other log messages (though there aren't many of those - I don't know
>> if you have more detail with log levels turned up to debug).  I wouldn't expect the webapp to require new serial numbers - no new bioentities or clobs are
>> being written and bags/precomputes are identified by unique table names using the separate objectstore_unique_integer sequence rather than serial.
>>
>> But clearly something is happening, perhaps (judging from my log) something periodic that is accidentally requesting IDs but doesn't actually need them.
> Hi Justin,
>
> I just fired up the web app in the debugger and was looking to see where I hit a breakpoint on getSerialWithConnection. I did not hit the breakpoint on startup,
> but did when I tried to save a list. After pasting in 10 gene identifiers, I hit it after hitting the 'save a list of 10 genes'  The stack trace showed that
> InterMineBag.init() calls ObjectStore.createObjectStoreBag() and that creates a new ObjectStoreBag() with an argument getSerial().intValue()

Yes sorry, I was wrong in my statement about osbags above.  Objectstore bags do get serial number IDs in table osbag_int, as they store bioentities by
objectstore ID for a list, whereas the userprofile lists (table bagvalues) are stored by gene/protein/etc. identifier.  I believe one reason for this is so that
an objectstore reload does not automatically update a user's lists if gene/protein names changed.  Instead, the user has to go through an explicit conversion
process.

But I would be surprised if creation of lists could trigger objectstore id growth to such an extent.  You should only see the log the log message "Got new set
of serial numbers..." every 1,000,000 ids (ObjectStoreInterMineImpl.SEQUENCE_MULTIPLE).

I suspect that a much bigger issue here, afaics, is that the fact that on every restart of the ObjectStore, a new batch of SEQUENCE_MULTIPLE is taken on the
first ID request.  This will happen both on every integrate step, and on every restart of Tomcat.  This is because ObjectStoreInterMineImpl.sequenceOffset gets
reset to SEQUENCE_MULTIPLE on initialization.

One can see this by looking at any of the major object tables (gene, protein, etc.).  The IDs start very near a 1,000,000 round number (probably not exact due
to IDs grabbed for dataset/datasource entries, etc.).

A cheap hack to test this would be to set SEQUENCE_MULTIPLE to something like 1,000 instead of 1,000,000.  The performance impact is going to be practically
zero and should vastly reduce the ID gaps (I am be somewhat dubious whether InterMine's scheme of generating most IDs in memory has any significant performance
benefit in the first place).  Worth a try, Joe?

>
> The current state of my mine is that I'm about 3/4 the way through my id's. Here is the "histogram" of my intermineobject id's:
>
> phytomine-11.0.5=# select 100000000*((id/100000000)::int),count(*) from intermineobject group by 1 order by 1;
>    ?column?   |   count
> -------------+-----------
>   -2100000000 |  22041400
>   -1400000000 |         2
>   -1000000000 |   9002637
>    -800000000 |  25893789
>    -700000000 |   8123453
>             0 |  50039666
>     100000000 |  63311389
>     200000000 |  62498622
>     300000000 |  64681926
>     400000000 | 100000000
>     500000000 |  99093219
>     600000000 |  98746936
>     700000000 |  98667077
>     800000000 | 100000000
>     900000000 |  95211450
>    1000000000 | 100000000
>    1100000000 |  52068134
>    1200000000 |  52786551
>    1300000000 |  56096351
>    1400000000 |  35956705
>    2100000000 |    738133
> (21 rows)
>
> The big jump from 1.4G to 2.1G was after the second loading process. Then the id wrapped and went negative. A second and third big jump happened in the third
> and fourth loading process. We currently have ~500 saved lists in the user profile database that were created after this mine build. What I'm seeing in the gaps
> indicate it should be more like 1200 lists; maybe there were lists created but never saved.

If the InterMineBag.init() code wasn't triggered until you actually tried to save the list then creation of lists in itself shouldn't allocate an objectstore
ID.  However, I can imagine that there may be other code paths that are allocating an ID without ever actually needing it.  This is speculation, I don't know
for sure at this point.

>
> What is flymine like? How many lists are created between builds of the mine? And do you see issues of the id wrapping around?

I don't have any experience of Flymine so Julie would have to answer this but it would be a good data point.

>
> By the way, I did need to make a few changes to get the web app to report objects with negative id's. In
> org.intermine.web.struts.ReportController.getRequestedObject, for example, the call to StringUtils.isNumeric returns false of the argument is a negative integer.
>
> I'm OK for now. And I think that I'm going to do a full rebuild before the id counter fully wraps around. It is nonetheless a concern that the serial is getting
> incremented and not being used.

I still get queasy at the idea of wrapping to negative IDs like this.  But 64 bit is also a considerably undertaking with definite storage and possible
processing impact.  imo, we need to start looking at your existing work in this area.  But then, if we go to a different backend technology (e.g. graph DB) then
that is wasted.

Another alternative is to use Guids instead of integers for IDs.  But this would expand storage requirements even further with the current database architecture.

>
> Thanks,
>
> Joe
>>
>> I did make a small change to the serial number message in github dev just to show which sequence number is being taken.  But to actually pin down what is
>> happening could you temporarily insert a stack trace log when the new sequence number is being requested? That would hopefully give us a toehold on the code
>> path making these requests.
>>
>> --
>> Justin Clark-Casey, Synbiomine/InterMine Developer
>> http://synbiomine.org
>> http://twitter.com/justincc
>>
>> On 03/06/16 23:34, Joe Carlson wrote:
>>> Hello,
>>>
>>> I'm getting more and more concerned with running out of id’s in the database. To this end I'm presently working on modifications to the codebase to
>>> optionally allow for 64-bit ids (I can load and post-process malariamine; there are a few other issues to get the webapp up.) More on that later.
>>>
>>> In the meantime, I've noticed another issue that has caused my id counter to go negative. I can run through id’s when the web app is up. I see the message in
>>> the log file saying “Got a new set of serial number” from ObjectStoreInterMineImpl in the tomcat log file. What triggers this? Does it happen with every web
>>> app restart? Is this used in storing lists/precomputes/query batching?
>>>
>>> For our past couple of minor releases, I’ve been adding data to the previous version. (I start by restoring the db dump on a new build system, add data to it
>>> and then switch over the public site to that.) Now I see that I have huge gaps in the ids corresponding to the time that the web app was up. Big gaps of
>>> 700,000,000.
>>>
>>> In retrospect I should have checked the value of the sequence before entering more data into the mine. That would have prevented the holes and the fact that
>>> my id’s are now negative. But beside that, it’s not unreasonable to expect that a web app could run through all ids with enough restarts.
>>>
>>> Any thoughts on this? What is the serial numbers used for in the web app?
>>>
>>> Thanks,
>>>
>>> Joe Carlson
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: burning through id's

joe carlson


On 06/14/2016 09:23 AM, Justin Clark-Casey wrote:

> On 13/06/16 20:13, Joe Carlson wrote:
>>
>>
>> On 06/07/2016 10:53 AM, Justin Clark-Casey wrote:
>>> Hi Joe.  I don't see this happening on a fresh restart of Synbiomine
>>> but I did see it happening at points in the log of a previous
>>> longer-running instance.
>>>
>>> In my case, the messages about serial number generation are not
>>> obviously related to any other log messages (though there aren't
>>> many of those - I don't know
>>> if you have more detail with log levels turned up to debug). I
>>> wouldn't expect the webapp to require new serial numbers - no new
>>> bioentities or clobs are
>>> being written and bags/precomputes are identified by unique table
>>> names using the separate objectstore_unique_integer sequence rather
>>> than serial.
>>>
>>> But clearly something is happening, perhaps (judging from my log)
>>> something periodic that is accidentally requesting IDs but doesn't
>>> actually need them.
>> Hi Justin,
>>
>> I just fired up the web app in the debugger and was looking to see
>> where I hit a breakpoint on getSerialWithConnection. I did not hit
>> the breakpoint on startup,
>> but did when I tried to save a list. After pasting in 10 gene
>> identifiers, I hit it after hitting the 'save a list of 10 genes'  
>> The stack trace showed that
>> InterMineBag.init() calls ObjectStore.createObjectStoreBag() and that
>> creates a new ObjectStoreBag() with an argument getSerial().intValue()
>
> Yes sorry, I was wrong in my statement about osbags above. Objectstore
> bags do get serial number IDs in table osbag_int, as they store
> bioentities by objectstore ID for a list, whereas the userprofile
> lists (table bagvalues) are stored by gene/protein/etc. identifier.  I
> believe one reason for this is so that an objectstore reload does not
> automatically update a user's lists if gene/protein names changed.  
> Instead, the user has to go through an explicit conversion process.

yeah. forgot about the os_bagint table. And looking at that histogram of
id's, I can find the holes in my intermineobject histogram:

select 100000000*((bagid/100000000)::int),count(*) from osbag_int group
by 1 order by 1;
   ?column?   |  count
-------------+---------
  -2100000000 |   45961
  -2000000000 |  111849
  -1900000000 |  165705
  -1800000000 |   56501
  -1700000000 |   49191
  -1600000000 |   27688
  -1500000000 |   58050
  -1400000000 |  824738
  -1300000000 |  733913
  -1200000000 |  274498
  -1100000000 |  114568
  -1000000000 |  199768
   -900000000 |   57218
   -800000000 |   33912
   -700000000 |   52963
   -600000000 |  120512
   -500000000 |   13683
   -400000000 |   99966
   -300000000 |   56429
   -200000000 |   37181
   1400000000 |    1116
   1500000000 |   56400
   1600000000 |   46187
   1700000000 |   34210
   1800000000 |   54825
   1900000000 |   48696
   2000000000 | 1108096
   2100000000 |   80196
(28 rows)

>
> But I would be surprised if creation of lists could trigger
> objectstore id growth to such an extent.  You should only see the log
> the log message "Got new set of serial numbers..." every 1,000,000 ids
> (ObjectStoreInterMineImpl.SEQUENCE_MULTIPLE).
>
> I suspect that a much bigger issue here, afaics, is that the fact that
> on every restart of the ObjectStore, a new batch of SEQUENCE_MULTIPLE
> is taken on the first ID request.  This will happen both on every
> integrate step, and on every restart of Tomcat.  This is because
> ObjectStoreInterMineImpl.sequenceOffset gets reset to
> SEQUENCE_MULTIPLE on initialization.
>
> One can see this by looking at any of the major object tables (gene,
> protein, etc.).  The IDs start very near a 1,000,000 round number
> (probably not exact due to IDs grabbed for dataset/datasource entries,
> etc.).
>
> A cheap hack to test this would be to set SEQUENCE_MULTIPLE to
> something like 1,000 instead of 1,000,000.  The performance impact is
> going to be practically zero and should vastly reduce the ID gaps (I
> am be somewhat dubious whether InterMine's scheme of generating most
> IDs in memory has any significant performance benefit in the first
> place).  Worth a try, Joe?

worth a try. But, honestly, it would not be a help for very long. The id
really, really needs to be a 64 bit int.

>
>>
>> The current state of my mine is that I'm about 3/4 the way through my
>> id's. Here is the "histogram" of my intermineobject id's:
>>
>> phytomine-11.0.5=# select 100000000*((id/100000000)::int),count(*)
>> from intermineobject group by 1 order by 1;
>>    ?column?   |   count
>> -------------+-----------
>>   -2100000000 |  22041400
>>   -1400000000 |         2
>>   -1000000000 |   9002637
>>    -800000000 |  25893789
>>    -700000000 |   8123453
>>             0 |  50039666
>>     100000000 |  63311389
>>     200000000 |  62498622
>>     300000000 |  64681926
>>     400000000 | 100000000
>>     500000000 |  99093219
>>     600000000 |  98746936
>>     700000000 |  98667077
>>     800000000 | 100000000
>>     900000000 |  95211450
>>    1000000000 | 100000000
>>    1100000000 |  52068134
>>    1200000000 |  52786551
>>    1300000000 |  56096351
>>    1400000000 |  35956705
>>    2100000000 |    738133
>> (21 rows)
>>
>> The big jump from 1.4G to 2.1G was after the second loading process.
>> Then the id wrapped and went negative. A second and third big jump
>> happened in the third
>> and fourth loading process. We currently have ~500 saved lists in the
>> user profile database that were created after this mine build. What
>> I'm seeing in the gaps
>> indicate it should be more like 1200 lists; maybe there were lists
>> created but never saved.
>
> If the InterMineBag.init() code wasn't triggered until you actually
> tried to save the list then creation of lists in itself shouldn't
> allocate an objectstore ID.  However, I can imagine that there may be
> other code paths that are allocating an ID without ever actually
> needing it.  This is speculation, I don't know for sure at this point.
>
>>
>> What is flymine like? How many lists are created between builds of
>> the mine? And do you see issues of the id wrapping around?
>
> I don't have any experience of Flymine so Julie would have to answer
> this but it would be a good data point.
>
>>
>> By the way, I did need to make a few changes to get the web app to
>> report objects with negative id's. In
>> org.intermine.web.struts.ReportController.getRequestedObject, for
>> example, the call to StringUtils.isNumeric returns false of the
>> argument is a negative integer.
>>
>> I'm OK for now. And I think that I'm going to do a full rebuild
>> before the id counter fully wraps around. It is nonetheless a concern
>> that the serial is getting
>> incremented and not being used.
>
> I still get queasy at the idea of wrapping to negative IDs like this.  
> But 64 bit is also a considerably undertaking with definite storage
> and possible processing impact.  imo, we need to start looking at your
> existing work in this area.  But then, if we go to a different backend
> technology (e.g. graph DB) then that is wasted.
>
> Another alternative is to use Guids instead of integers for IDs. But
> this would expand storage requirements even further with the current
> database architecture.


I'm still taking a stab at the 64 bit id problem. What I have so far is
that I'm using a wrapper class for the id. There was a lot of autoboxing
in the code that I needed to track down and replace, but I'm close to
having something usable. For now, the wrapper class is still a 32 bit
integer. If I can get this code to work then I ought to be able to drop
in a 64 bit integer. And, I hope, I can make it configurable at build
time so there need not be a storage hit for sites that don't need it.

Joe

>
>>
>> Thanks,
>>
>> Joe
>>>
>>> I did make a small change to the serial number message in github dev
>>> just to show which sequence number is being taken. But to actually
>>> pin down what is
>>> happening could you temporarily insert a stack trace log when the
>>> new sequence number is being requested? That would hopefully give us
>>> a toehold on the code
>>> path making these requests.
>>>
>>> --
>>> Justin Clark-Casey, Synbiomine/InterMine Developer
>>> http://synbiomine.org
>>> http://twitter.com/justincc
>>>
>>> On 03/06/16 23:34, Joe Carlson wrote:
>>>> Hello,
>>>>
>>>> I'm getting more and more concerned with running out of id’s in the
>>>> database. To this end I'm presently working on modifications to the
>>>> codebase to
>>>> optionally allow for 64-bit ids (I can load and post-process
>>>> malariamine; there are a few other issues to get the webapp up.)
>>>> More on that later.
>>>>
>>>> In the meantime, I've noticed another issue that has caused my id
>>>> counter to go negative. I can run through id’s when the web app is
>>>> up. I see the message in
>>>> the log file saying “Got a new set of serial number” from
>>>> ObjectStoreInterMineImpl in the tomcat log file. What triggers
>>>> this? Does it happen with every web
>>>> app restart? Is this used in storing lists/precomputes/query batching?
>>>>
>>>> For our past couple of minor releases, I’ve been adding data to the
>>>> previous version. (I start by restoring the db dump on a new build
>>>> system, add data to it
>>>> and then switch over the public site to that.) Now I see that I
>>>> have huge gaps in the ids corresponding to the time that the web
>>>> app was up. Big gaps of
>>>> 700,000,000.
>>>>
>>>> In retrospect I should have checked the value of the sequence
>>>> before entering more data into the mine. That would have prevented
>>>> the holes and the fact that
>>>> my id’s are now negative. But beside that, it’s not unreasonable to
>>>> expect that a web app could run through all ids with enough restarts.
>>>>
>>>> Any thoughts on this? What is the serial numbers used for in the
>>>> web app?
>>>>
>>>> Thanks,
>>>>
>>>> Joe Carlson
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: burning through id's

joe carlson
In reply to this post by Justin Clark-Casey-2
Hi Justin,

After hitting 'send', I started wondering about the bag ids.

Is is really necessary for there to be no overlap between id's in the
intermineobject table and bagid's in the osbag_int table?

the bagid is used in mapping the savedbag from the userprofile database
to contents of the bag in a particular mine. I understand that part of
the structure. But it's not clear that this has to have a distinct value
from all other intermine objects. In that case, creating a separate
serial for the bag id and using that when creating bags could help.

(I realize that part of the problem is the fact that I'm adding to a
mine after it has been deployed was probably not a use case you had
anticipated when this was designed.)

Have I missed something?

Joe

On 06/14/2016 09:23 AM, Justin Clark-Casey wrote:

> On 13/06/16 20:13, Joe Carlson wrote:
>>
>>
>> On 06/07/2016 10:53 AM, Justin Clark-Casey wrote:
>>> Hi Joe.  I don't see this happening on a fresh restart of Synbiomine
>>> but I did see it happening at points in the log of a previous
>>> longer-running instance.
>>>
>>> In my case, the messages about serial number generation are not
>>> obviously related to any other log messages (though there aren't
>>> many of those - I don't know
>>> if you have more detail with log levels turned up to debug). I
>>> wouldn't expect the webapp to require new serial numbers - no new
>>> bioentities or clobs are
>>> being written and bags/precomputes are identified by unique table
>>> names using the separate objectstore_unique_integer sequence rather
>>> than serial.
>>>
>>> But clearly something is happening, perhaps (judging from my log)
>>> something periodic that is accidentally requesting IDs but doesn't
>>> actually need them.
>> Hi Justin,
>>
>> I just fired up the web app in the debugger and was looking to see
>> where I hit a breakpoint on getSerialWithConnection. I did not hit
>> the breakpoint on startup,
>> but did when I tried to save a list. After pasting in 10 gene
>> identifiers, I hit it after hitting the 'save a list of 10 genes'  
>> The stack trace showed that
>> InterMineBag.init() calls ObjectStore.createObjectStoreBag() and that
>> creates a new ObjectStoreBag() with an argument getSerial().intValue()
>
> Yes sorry, I was wrong in my statement about osbags above. Objectstore
> bags do get serial number IDs in table osbag_int, as they store
> bioentities by objectstore ID for a list, whereas the userprofile
> lists (table bagvalues) are stored by gene/protein/etc. identifier.  I
> believe one reason for this is so that an objectstore reload does not
> automatically update a user's lists if gene/protein names changed.  
> Instead, the user has to go through an explicit conversion process.
>
> But I would be surprised if creation of lists could trigger
> objectstore id growth to such an extent.  You should only see the log
> the log message "Got new set of serial numbers..." every 1,000,000 ids
> (ObjectStoreInterMineImpl.SEQUENCE_MULTIPLE).
>
> I suspect that a much bigger issue here, afaics, is that the fact that
> on every restart of the ObjectStore, a new batch of SEQUENCE_MULTIPLE
> is taken on the first ID request.  This will happen both on every
> integrate step, and on every restart of Tomcat.  This is because
> ObjectStoreInterMineImpl.sequenceOffset gets reset to
> SEQUENCE_MULTIPLE on initialization.
>
> One can see this by looking at any of the major object tables (gene,
> protein, etc.).  The IDs start very near a 1,000,000 round number
> (probably not exact due to IDs grabbed for dataset/datasource entries,
> etc.).
>
> A cheap hack to test this would be to set SEQUENCE_MULTIPLE to
> something like 1,000 instead of 1,000,000.  The performance impact is
> going to be practically zero and should vastly reduce the ID gaps (I
> am be somewhat dubious whether InterMine's scheme of generating most
> IDs in memory has any significant performance benefit in the first
> place).  Worth a try, Joe?
>
>>
>> The current state of my mine is that I'm about 3/4 the way through my
>> id's. Here is the "histogram" of my intermineobject id's:
>>
>> phytomine-11.0.5=# select 100000000*((id/100000000)::int),count(*)
>> from intermineobject group by 1 order by 1;
>>    ?column?   |   count
>> -------------+-----------
>>   -2100000000 |  22041400
>>   -1400000000 |         2
>>   -1000000000 |   9002637
>>    -800000000 |  25893789
>>    -700000000 |   8123453
>>             0 |  50039666
>>     100000000 |  63311389
>>     200000000 |  62498622
>>     300000000 |  64681926
>>     400000000 | 100000000
>>     500000000 |  99093219
>>     600000000 |  98746936
>>     700000000 |  98667077
>>     800000000 | 100000000
>>     900000000 |  95211450
>>    1000000000 | 100000000
>>    1100000000 |  52068134
>>    1200000000 |  52786551
>>    1300000000 |  56096351
>>    1400000000 |  35956705
>>    2100000000 |    738133
>> (21 rows)
>>
>> The big jump from 1.4G to 2.1G was after the second loading process.
>> Then the id wrapped and went negative. A second and third big jump
>> happened in the third
>> and fourth loading process. We currently have ~500 saved lists in the
>> user profile database that were created after this mine build. What
>> I'm seeing in the gaps
>> indicate it should be more like 1200 lists; maybe there were lists
>> created but never saved.
>
> If the InterMineBag.init() code wasn't triggered until you actually
> tried to save the list then creation of lists in itself shouldn't
> allocate an objectstore ID.  However, I can imagine that there may be
> other code paths that are allocating an ID without ever actually
> needing it.  This is speculation, I don't know for sure at this point.
>
>>
>> What is flymine like? How many lists are created between builds of
>> the mine? And do you see issues of the id wrapping around?
>
> I don't have any experience of Flymine so Julie would have to answer
> this but it would be a good data point.
>
>>
>> By the way, I did need to make a few changes to get the web app to
>> report objects with negative id's. In
>> org.intermine.web.struts.ReportController.getRequestedObject, for
>> example, the call to StringUtils.isNumeric returns false of the
>> argument is a negative integer.
>>
>> I'm OK for now. And I think that I'm going to do a full rebuild
>> before the id counter fully wraps around. It is nonetheless a concern
>> that the serial is getting
>> incremented and not being used.
>
> I still get queasy at the idea of wrapping to negative IDs like this.  
> But 64 bit is also a considerably undertaking with definite storage
> and possible processing impact.  imo, we need to start looking at your
> existing work in this area.  But then, if we go to a different backend
> technology (e.g. graph DB) then that is wasted.
>
> Another alternative is to use Guids instead of integers for IDs. But
> this would expand storage requirements even further with the current
> database architecture.
>
>>
>> Thanks,
>>
>> Joe
>>>
>>> I did make a small change to the serial number message in github dev
>>> just to show which sequence number is being taken. But to actually
>>> pin down what is
>>> happening could you temporarily insert a stack trace log when the
>>> new sequence number is being requested? That would hopefully give us
>>> a toehold on the code
>>> path making these requests.
>>>
>>> --
>>> Justin Clark-Casey, Synbiomine/InterMine Developer
>>> http://synbiomine.org
>>> http://twitter.com/justincc
>>>
>>> On 03/06/16 23:34, Joe Carlson wrote:
>>>> Hello,
>>>>
>>>> I'm getting more and more concerned with running out of id’s in the
>>>> database. To this end I'm presently working on modifications to the
>>>> codebase to
>>>> optionally allow for 64-bit ids (I can load and post-process
>>>> malariamine; there are a few other issues to get the webapp up.)
>>>> More on that later.
>>>>
>>>> In the meantime, I've noticed another issue that has caused my id
>>>> counter to go negative. I can run through id’s when the web app is
>>>> up. I see the message in
>>>> the log file saying “Got a new set of serial number” from
>>>> ObjectStoreInterMineImpl in the tomcat log file. What triggers
>>>> this? Does it happen with every web
>>>> app restart? Is this used in storing lists/precomputes/query batching?
>>>>
>>>> For our past couple of minor releases, I’ve been adding data to the
>>>> previous version. (I start by restoring the db dump on a new build
>>>> system, add data to it
>>>> and then switch over the public site to that.) Now I see that I
>>>> have huge gaps in the ids corresponding to the time that the web
>>>> app was up. Big gaps of
>>>> 700,000,000.
>>>>
>>>> In retrospect I should have checked the value of the sequence
>>>> before entering more data into the mine. That would have prevented
>>>> the holes and the fact that
>>>> my id’s are now negative. But beside that, it’s not unreasonable to
>>>> expect that a web app could run through all ids with enough restarts.
>>>>
>>>> Any thoughts on this? What is the serial numbers used for in the
>>>> web app?
>>>>
>>>> Thanks,
>>>>
>>>> Joe Carlson
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: burning through id's

Justin Clark-Casey-2
That's a good point, Joe.  I don't know of any reason why the bags couldn't have their own ID sequence rather than taking it from the main serial sequence.

A cheap way to test this would be to use the existing objectstore_unique_integer sequence, which is already used for temporary bags amongst other temporary
tables.  This could be as simple as having the bag creation code call ObjectStoreInterMineImpl.getUniqueInteger() rather than getSerial().

Ultimately, I suppose one would create a new sequence by plumbing the creation in at BuildDbTask.execute() and the removal on DB refresh at the top of execute().

On a side note, I just fixed an omission here in 1ec6ce1 dev where the serial sequence was removed before recreation but not objectstore_unique_integer.  The
code in execute() is also pretty horrible, with some copy/pasting of the sequence code.  At some point this needs to be extracted into a method, even more so if
it's going to be executed more than twice.

On 14/06/16 17:57, Joe Carlson wrote:

> Hi Justin,
>
> After hitting 'send', I started wondering about the bag ids.
>
> Is is really necessary for there to be no overlap between id's in the intermineobject table and bagid's in the osbag_int table?
>
> the bagid is used in mapping the savedbag from the userprofile database to contents of the bag in a particular mine. I understand that part of the structure.
> But it's not clear that this has to have a distinct value from all other intermine objects. In that case, creating a separate serial for the bag id and using
> that when creating bags could help.
>
> (I realize that part of the problem is the fact that I'm adding to a mine after it has been deployed was probably not a use case you had anticipated when this
> was designed.)
>
> Have I missed something?
>
> Joe
>
> On 06/14/2016 09:23 AM, Justin Clark-Casey wrote:
>> On 13/06/16 20:13, Joe Carlson wrote:
>>>
>>>
>>> On 06/07/2016 10:53 AM, Justin Clark-Casey wrote:
>>>> Hi Joe.  I don't see this happening on a fresh restart of Synbiomine but I did see it happening at points in the log of a previous longer-running instance.
>>>>
>>>> In my case, the messages about serial number generation are not obviously related to any other log messages (though there aren't many of those - I don't know
>>>> if you have more detail with log levels turned up to debug). I wouldn't expect the webapp to require new serial numbers - no new bioentities or clobs are
>>>> being written and bags/precomputes are identified by unique table names using the separate objectstore_unique_integer sequence rather than serial.
>>>>
>>>> But clearly something is happening, perhaps (judging from my log) something periodic that is accidentally requesting IDs but doesn't actually need them.
>>> Hi Justin,
>>>
>>> I just fired up the web app in the debugger and was looking to see where I hit a breakpoint on getSerialWithConnection. I did not hit the breakpoint on startup,
>>> but did when I tried to save a list. After pasting in 10 gene identifiers, I hit it after hitting the 'save a list of 10 genes' The stack trace showed that
>>> InterMineBag.init() calls ObjectStore.createObjectStoreBag() and that creates a new ObjectStoreBag() with an argument getSerial().intValue()
>>
>> Yes sorry, I was wrong in my statement about osbags above. Objectstore bags do get serial number IDs in table osbag_int, as they store bioentities by
>> objectstore ID for a list, whereas the userprofile lists (table bagvalues) are stored by gene/protein/etc. identifier.  I believe one reason for this is so
>> that an objectstore reload does not automatically update a user's lists if gene/protein names changed. Instead, the user has to go through an explicit
>> conversion process.
>>
>> But I would be surprised if creation of lists could trigger objectstore id growth to such an extent.  You should only see the log the log message "Got new set
>> of serial numbers..." every 1,000,000 ids (ObjectStoreInterMineImpl.SEQUENCE_MULTIPLE).
>>
>> I suspect that a much bigger issue here, afaics, is that the fact that on every restart of the ObjectStore, a new batch of SEQUENCE_MULTIPLE is taken on the
>> first ID request.  This will happen both on every integrate step, and on every restart of Tomcat.  This is because ObjectStoreInterMineImpl.sequenceOffset
>> gets reset to SEQUENCE_MULTIPLE on initialization.
>>
>> One can see this by looking at any of the major object tables (gene, protein, etc.).  The IDs start very near a 1,000,000 round number (probably not exact due
>> to IDs grabbed for dataset/datasource entries, etc.).
>>
>> A cheap hack to test this would be to set SEQUENCE_MULTIPLE to something like 1,000 instead of 1,000,000.  The performance impact is going to be practically
>> zero and should vastly reduce the ID gaps (I am be somewhat dubious whether InterMine's scheme of generating most IDs in memory has any significant
>> performance benefit in the first place).  Worth a try, Joe?
>>
>>>
>>> The current state of my mine is that I'm about 3/4 the way through my id's. Here is the "histogram" of my intermineobject id's:
>>>
>>> phytomine-11.0.5=# select 100000000*((id/100000000)::int),count(*) from intermineobject group by 1 order by 1;
>>>    ?column?   |   count
>>> -------------+-----------
>>>   -2100000000 |  22041400
>>>   -1400000000 |         2
>>>   -1000000000 |   9002637
>>>    -800000000 |  25893789
>>>    -700000000 |   8123453
>>>             0 |  50039666
>>>     100000000 |  63311389
>>>     200000000 |  62498622
>>>     300000000 |  64681926
>>>     400000000 | 100000000
>>>     500000000 |  99093219
>>>     600000000 |  98746936
>>>     700000000 |  98667077
>>>     800000000 | 100000000
>>>     900000000 |  95211450
>>>    1000000000 | 100000000
>>>    1100000000 |  52068134
>>>    1200000000 |  52786551
>>>    1300000000 |  56096351
>>>    1400000000 |  35956705
>>>    2100000000 |    738133
>>> (21 rows)
>>>
>>> The big jump from 1.4G to 2.1G was after the second loading process. Then the id wrapped and went negative. A second and third big jump happened in the third
>>> and fourth loading process. We currently have ~500 saved lists in the user profile database that were created after this mine build. What I'm seeing in the gaps
>>> indicate it should be more like 1200 lists; maybe there were lists created but never saved.
>>
>> If the InterMineBag.init() code wasn't triggered until you actually tried to save the list then creation of lists in itself shouldn't allocate an objectstore
>> ID.  However, I can imagine that there may be other code paths that are allocating an ID without ever actually needing it.  This is speculation, I don't know
>> for sure at this point.
>>
>>>
>>> What is flymine like? How many lists are created between builds of the mine? And do you see issues of the id wrapping around?
>>
>> I don't have any experience of Flymine so Julie would have to answer this but it would be a good data point.
>>
>>>
>>> By the way, I did need to make a few changes to get the web app to report objects with negative id's. In
>>> org.intermine.web.struts.ReportController.getRequestedObject, for example, the call to StringUtils.isNumeric returns false of the argument is a negative
>>> integer.
>>>
>>> I'm OK for now. And I think that I'm going to do a full rebuild before the id counter fully wraps around. It is nonetheless a concern that the serial is getting
>>> incremented and not being used.
>>
>> I still get queasy at the idea of wrapping to negative IDs like this. But 64 bit is also a considerably undertaking with definite storage and possible
>> processing impact.  imo, we need to start looking at your existing work in this area.  But then, if we go to a different backend technology (e.g. graph DB)
>> then that is wasted.
>>
>> Another alternative is to use Guids instead of integers for IDs. But this would expand storage requirements even further with the current database architecture.
>>
>>>
>>> Thanks,
>>>
>>> Joe
>>>>
>>>> I did make a small change to the serial number message in github dev just to show which sequence number is being taken. But to actually pin down what is
>>>> happening could you temporarily insert a stack trace log when the new sequence number is being requested? That would hopefully give us a toehold on the code
>>>> path making these requests.
>>>>
>>>> --
>>>> Justin Clark-Casey, Synbiomine/InterMine Developer
>>>> http://synbiomine.org
>>>> http://twitter.com/justincc
>>>>
>>>> On 03/06/16 23:34, Joe Carlson wrote:
>>>>> Hello,
>>>>>
>>>>> I'm getting more and more concerned with running out of id’s in the database. To this end I'm presently working on modifications to the codebase to
>>>>> optionally allow for 64-bit ids (I can load and post-process malariamine; there are a few other issues to get the webapp up.) More on that later.
>>>>>
>>>>> In the meantime, I've noticed another issue that has caused my id counter to go negative. I can run through id’s when the web app is up. I see the message in
>>>>> the log file saying “Got a new set of serial number” from ObjectStoreInterMineImpl in the tomcat log file. What triggers this? Does it happen with every web
>>>>> app restart? Is this used in storing lists/precomputes/query batching?
>>>>>
>>>>> For our past couple of minor releases, I’ve been adding data to the previous version. (I start by restoring the db dump on a new build system, add data to it
>>>>> and then switch over the public site to that.) Now I see that I have huge gaps in the ids corresponding to the time that the web app was up. Big gaps of
>>>>> 700,000,000.
>>>>>
>>>>> In retrospect I should have checked the value of the sequence before entering more data into the mine. That would have prevented the holes and the fact that
>>>>> my id’s are now negative. But beside that, it’s not unreasonable to expect that a web app could run through all ids with enough restarts.
>>>>>
>>>>> Any thoughts on this? What is the serial numbers used for in the web app?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Joe Carlson
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> [hidden email]
>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Loading...