Custom DB incorporation in Intermine

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Custom DB incorporation in Intermine

Selvaraj, Balamuruganand
Hi Julie / Richard,
 
I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
 
1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?
 
2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.
 
Thanks for your reply.
 
Best regards,
Bala
 

_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Custom DB incorporation in Intermine

Richard Smith
Hi Bala,

Selvaraj, Balamuruganand wrote:
> Hi Julie / Richard,
>
> I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
>
> 1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
> Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?

Is this a local database other than chado?  If reading from a chado
database then it's best to build on the existing chado converter and
processors.

If you're reading from a different database you can use the make_source
script to create a new 'db' source:

bio/scripts/make_source test db

Which will create a bio/sources/test directory including a TestConverter
which extends BioDBConverter.  All of the parsers extend a DataConverter
class with specific extensions for reading files, directories, DBs or
Items XML.

Parsers are called by including the source in project.xml.  In this case
you would add 'test' with an attribute to identifier the database:

   <property name="source.db.name" value="test-db"/>

You set up database access properties for test-db in your
.intermine/xxmine.properties file, just as for chado database:

db.test.datasource.serverName=servername
db.test.datasource.databaseName=database-name
db.test.datasource.user=user
db.test.datasource.password=password

db.test.datasource.class=org.postgresql.jdbc3.Jdbc3PoolingDataSource
db.test.datasource.dataSourceName=db.test
db.test.datasource.maxConnections=10
db.test.driver=org.postgresql.Driver
db.test.platform=PostgreSQL


So the TestConverter can now be called as with other sources by running
 > ant -Dsource=test from the xxmine/integrate.  It has a Database object
and connection already set up.

You can read from the source database and create and store Items just
as with other types of source.  See an example, this reads from a file
but the item creation and store() method works exactly the same:

http://intrac.flymine.org/browser/branches/intermine_0_93/bio/sources/wormbase-identifiers/main/src/org/intermine/bio/dataconversion/WormBaseIdentifiersConverter.java



> 2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.


Do you mean if one object you store references another object?  There is
some information about how the mapping to the database works here:

http://www.intermine.org/wiki/GettingStarted#a6.2Creatingadatabase

For references to a single object there usually a foreign key in the
first table.  e.g. Gene has a reference called 'organism' to the
Organism class.  In the database the gene table has an organismid
column.

For collections there will be an additional indirection table created,
e.g. Gene has a collection of Protein objects, there will be an extra
genesproteins table created.


I hope this helps, please let me know if you have any further questions.

Regards,
Richard.





> Thanks for your reply.
>
> Best regards,
> Bala
>
>


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Custom DB incorporation in Intermine

Selvaraj, Balamuruganand
Richard,

Thanks for your reply.

The testConvertor was created without any issues. Could you tell me what I might have to add for these class level variables defined in the  testConvertor.java file.

    private static final String DATASET_TITLE = "Add DataSet.title here";
    private static final String DATA_SOURCE_NAME = "Add DataSource.name here";

I guess DATA_SOURCE_NAME = "test"
if I had defined <property name="source.db.name" value="test"/> in XML. Am I right ?
How about DATASET_TITLE? What values I am supposed to add for this variable ?

-Bala

________________________________________
From: Richard Smith [[hidden email]]
Sent: Monday, June 07, 2010 5:16 AM
To: Selvaraj, Balamuruganand
Cc: [hidden email]; [hidden email]
Subject: Re: Custom DB incorporation in Intermine

Hi Bala,

Selvaraj, Balamuruganand wrote:
> Hi Julie / Richard,
>
> I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
>
> 1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
> Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?

Is this a local database other than chado?  If reading from a chado
database then it's best to build on the existing chado converter and
processors.

If you're reading from a different database you can use the make_source
script to create a new 'db' source:

bio/scripts/make_source test db

Which will create a bio/sources/test directory including a TestConverter
which extends BioDBConverter.  All of the parsers extend a DataConverter
class with specific extensions for reading files, directories, DBs or
Items XML.

Parsers are called by including the source in project.xml.  In this case
you would add 'test' with an attribute to identifier the database:

   <property name="source.db.name" value="test-db"/>

You set up database access properties for test-db in your
.intermine/xxmine.properties file, just as for chado database:

db.test.datasource.serverName=servername
db.test.datasource.databaseName=database-name
db.test.datasource.user=user
db.test.datasource.password=password

db.test.datasource.class=org.postgresql.jdbc3.Jdbc3PoolingDataSource
db.test.datasource.dataSourceName=db.test
db.test.datasource.maxConnections=10
db.test.driver=org.postgresql.Driver
db.test.platform=PostgreSQL


So the TestConverter can now be called as with other sources by running
 > ant -Dsource=test from the xxmine/integrate.  It has a Database object
and connection already set up.

You can read from the source database and create and store Items just
as with other types of source.  See an example, this reads from a file
but the item creation and store() method works exactly the same:

http://intrac.flymine.org/browser/branches/intermine_0_93/bio/sources/wormbase-identifiers/main/src/org/intermine/bio/dataconversion/WormBaseIdentifiersConverter.java



> 2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.


Do you mean if one object you store references another object?  There is
some information about how the mapping to the database works here:

http://www.intermine.org/wiki/GettingStarted#a6.2Creatingadatabase

For references to a single object there usually a foreign key in the
first table.  e.g. Gene has a reference called 'organism' to the
Organism class.  In the database the gene table has an organismid
column.

For collections there will be an additional indirection table created,
e.g. Gene has a collection of Protein objects, there will be an extra
genesproteins table created.


I hope this helps, please let me know if you have any further questions.

Regards,
Richard.





> Thanks for your reply.
>
> Best regards,
> Bala
>
>
_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Custom DB incorporation in Intermine

Selvaraj, Balamuruganand
Richard,

I have another question. When I run from the intergrate directory ant -Dsource=flybase-db, I get a Null pointer error as given below

java.lang.NullPointerException
  at org.intermine.task.Dependencies.followProjectDependencies(Dependencies.java:455)

However, I have defined the project dependencies in flybase/intergrate/project properties as
extra.project.dependencies = flybase/dbmodel

I am not sure as to why I am getting a Null pointer error. Can you suggest what I am doing wrong ?

Thanks.

Best regards,
Bala
________________________________________
From: Selvaraj, Balamuruganand
Sent: Monday, June 07, 2010 12:29 PM
To: Richard Smith
Cc: [hidden email]; [hidden email]; Goodman, Joshua Lawrence
Subject: RE: Custom DB incorporation in Intermine

Richard,

Thanks for your reply.

The testConvertor was created without any issues. Could you tell me what I might have to add for these class level variables defined in the  testConvertor.java file.

    private static final String DATASET_TITLE = "Add DataSet.title here";
    private static final String DATA_SOURCE_NAME = "Add DataSource.name here";

I guess DATA_SOURCE_NAME = "test"
if I had defined <property name="source.db.name" value="test"/> in XML. Am I right ?
How about DATASET_TITLE? What values I am supposed to add for this variable ?

-Bala

________________________________________
From: Richard Smith [[hidden email]]
Sent: Monday, June 07, 2010 5:16 AM
To: Selvaraj, Balamuruganand
Cc: [hidden email]; [hidden email]
Subject: Re: Custom DB incorporation in Intermine

Hi Bala,

Selvaraj, Balamuruganand wrote:
> Hi Julie / Richard,
>
> I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
>
> 1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
> Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?

Is this a local database other than chado?  If reading from a chado
database then it's best to build on the existing chado converter and
processors.

If you're reading from a different database you can use the make_source
script to create a new 'db' source:

bio/scripts/make_source test db

Which will create a bio/sources/test directory including a TestConverter
which extends BioDBConverter.  All of the parsers extend a DataConverter
class with specific extensions for reading files, directories, DBs or
Items XML.

Parsers are called by including the source in project.xml.  In this case
you would add 'test' with an attribute to identifier the database:

   <property name="source.db.name" value="test-db"/>

You set up database access properties for test-db in your
.intermine/xxmine.properties file, just as for chado database:

db.test.datasource.serverName=servername
db.test.datasource.databaseName=database-name
db.test.datasource.user=user
db.test.datasource.password=password

db.test.datasource.class=org.postgresql.jdbc3.Jdbc3PoolingDataSource
db.test.datasource.dataSourceName=db.test
db.test.datasource.maxConnections=10
db.test.driver=org.postgresql.Driver
db.test.platform=PostgreSQL


So the TestConverter can now be called as with other sources by running
 > ant -Dsource=test from the xxmine/integrate.  It has a Database object
and connection already set up.

You can read from the source database and create and store Items just
as with other types of source.  See an example, this reads from a file
but the item creation and store() method works exactly the same:

http://intrac.flymine.org/browser/branches/intermine_0_93/bio/sources/wormbase-identifiers/main/src/org/intermine/bio/dataconversion/WormBaseIdentifiersConverter.java



> 2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.


Do you mean if one object you store references another object?  There is
some information about how the mapping to the database works here:

http://www.intermine.org/wiki/GettingStarted#a6.2Creatingadatabase

For references to a single object there usually a foreign key in the
first table.  e.g. Gene has a reference called 'organism' to the
Organism class.  In the database the gene table has an organismid
column.

For collections there will be an additional indirection table created,
e.g. Gene has a collection of Protein objects, there will be an extra
genesproteins table created.


I hope this helps, please let me know if you have any further questions.

Regards,
Richard.





> Thanks for your reply.
>
> Best regards,
> Bala
>
>
_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Custom DB incorporation in Intermine

Richard Smith
Hi Bala,
Could you run the same command with the -v flag to get verbose output
and send the whole stack trace please?


If you created the flybase Mine with the make_mine script the properties
should have been set up correctly by default.  The flymine/integrate
project.properties look like this (minus commnets):


compile.dependencies = intermine/integrate/main,\
                        flymine/dbmodel,\
                        bio/core/main


intermine.properties.file = flymine.properties
default.intermine.properties.file =
../default.intermine.integrate.properties

extra.project.dependencies = flymine/dbmodel




Thanks,
Richard.



Selvaraj, Balamuruganand wrote:

> Richard,
>
> I have another question. When I run from the intergrate directory ant -Dsource=flybase-db, I get a Null pointer error as given below
>
> java.lang.NullPointerException
>   at org.intermine.task.Dependencies.followProjectDependencies(Dependencies.java:455)
>
> However, I have defined the project dependencies in flybase/intergrate/project properties as
> extra.project.dependencies = flybase/dbmodel
>
> I am not sure as to why I am getting a Null pointer error. Can you suggest what I am doing wrong ?
>
> Thanks.
>
> Best regards,
> Bala
> ________________________________________
> From: Selvaraj, Balamuruganand
> Sent: Monday, June 07, 2010 12:29 PM
> To: Richard Smith
> Cc: [hidden email]; [hidden email]; Goodman, Joshua Lawrence
> Subject: RE: Custom DB incorporation in Intermine
>
> Richard,
>
> Thanks for your reply.
>
> The testConvertor was created without any issues. Could you tell me what I might have to add for these class level variables defined in the  testConvertor.java file.
>
>     private static final String DATASET_TITLE = "Add DataSet.title here";
>     private static final String DATA_SOURCE_NAME = "Add DataSource.name here";
>
> I guess DATA_SOURCE_NAME = "test"
> if I had defined <property name="source.db.name" value="test"/> in XML. Am I right ?
> How about DATASET_TITLE? What values I am supposed to add for this variable ?
>
> -Bala
>
> ________________________________________
> From: Richard Smith [[hidden email]]
> Sent: Monday, June 07, 2010 5:16 AM
> To: Selvaraj, Balamuruganand
> Cc: [hidden email]; [hidden email]
> Subject: Re: Custom DB incorporation in Intermine
>
> Hi Bala,
>
> Selvaraj, Balamuruganand wrote:
>> Hi Julie / Richard,
>>
>> I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
>>
>> 1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
>> Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?
>
> Is this a local database other than chado?  If reading from a chado
> database then it's best to build on the existing chado converter and
> processors.
>
> If you're reading from a different database you can use the make_source
> script to create a new 'db' source:
>
> bio/scripts/make_source test db
>
> Which will create a bio/sources/test directory including a TestConverter
> which extends BioDBConverter.  All of the parsers extend a DataConverter
> class with specific extensions for reading files, directories, DBs or
> Items XML.
>
> Parsers are called by including the source in project.xml.  In this case
> you would add 'test' with an attribute to identifier the database:
>
>    <property name="source.db.name" value="test-db"/>
>
> You set up database access properties for test-db in your
> .intermine/xxmine.properties file, just as for chado database:
>
> db.test.datasource.serverName=servername
> db.test.datasource.databaseName=database-name
> db.test.datasource.user=user
> db.test.datasource.password=password
>
> db.test.datasource.class=org.postgresql.jdbc3.Jdbc3PoolingDataSource
> db.test.datasource.dataSourceName=db.test
> db.test.datasource.maxConnections=10
> db.test.driver=org.postgresql.Driver
> db.test.platform=PostgreSQL
>
>
> So the TestConverter can now be called as with other sources by running
>  > ant -Dsource=test from the xxmine/integrate.  It has a Database object
> and connection already set up.
>
> You can read from the source database and create and store Items just
> as with other types of source.  See an example, this reads from a file
> but the item creation and store() method works exactly the same:
>
> http://intrac.flymine.org/browser/branches/intermine_0_93/bio/sources/wormbase-identifiers/main/src/org/intermine/bio/dataconversion/WormBaseIdentifiersConverter.java
>
>
>
>> 2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.
>
>
> Do you mean if one object you store references another object?  There is
> some information about how the mapping to the database works here:
>
> http://www.intermine.org/wiki/GettingStarted#a6.2Creatingadatabase
>
> For references to a single object there usually a foreign key in the
> first table.  e.g. Gene has a reference called 'organism' to the
> Organism class.  In the database the gene table has an organismid
> column.
>
> For collections there will be an additional indirection table created,
> e.g. Gene has a collection of Protein objects, there will be an extra
> genesproteins table created.
>
>
> I hope this helps, please let me know if you have any further questions.
>
> Regards,
> Richard.
>
>
>
>
>
>> Thanks for your reply.
>>
>> Best regards,
>> Bala
>>
>>


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Custom DB incorporation in Intermine

Selvaraj, Balamuruganand
Richard,

Please find attached the Failed Build message that I got while running integrate. (Attachment as : flybase_intermine.log)

Yes, the project.properties seems to having all dependencies. But still I get this Null pointer error.
I have also attached the project.properties just for your reference.

Best regards,
Bala
________________________________________
From: Richard Smith [[hidden email]]
Sent: Monday, June 07, 2010 3:40 PM
To: Selvaraj, Balamuruganand
Cc: [hidden email]; [hidden email]; Goodman, Joshua Lawrence
Subject: Re: Custom DB incorporation in Intermine

Hi Bala,
Could you run the same command with the -v flag to get verbose output
and send the whole stack trace please?


If you created the flybase Mine with the make_mine script the properties
should have been set up correctly by default.  The flymine/integrate
project.properties look like this (minus commnets):


compile.dependencies = intermine/integrate/main,\
                        flymine/dbmodel,\
                        bio/core/main


intermine.properties.file = flymine.properties
default.intermine.properties.file =
../default.intermine.integrate.properties

extra.project.dependencies = flymine/dbmodel




Thanks,
Richard.



Selvaraj, Balamuruganand wrote:

> Richard,
>
> I have another question. When I run from the intergrate directory ant -Dsource=flybase-db, I get a Null pointer error as given below
>
> java.lang.NullPointerException
>   at org.intermine.task.Dependencies.followProjectDependencies(Dependencies.java:455)
>
> However, I have defined the project dependencies in flybase/intergrate/project properties as
> extra.project.dependencies = flybase/dbmodel
>
> I am not sure as to why I am getting a Null pointer error. Can you suggest what I am doing wrong ?
>
> Thanks.
>
> Best regards,
> Bala
> ________________________________________
> From: Selvaraj, Balamuruganand
> Sent: Monday, June 07, 2010 12:29 PM
> To: Richard Smith
> Cc: [hidden email]; [hidden email]; Goodman, Joshua Lawrence
> Subject: RE: Custom DB incorporation in Intermine
>
> Richard,
>
> Thanks for your reply.
>
> The testConvertor was created without any issues. Could you tell me what I might have to add for these class level variables defined in the  testConvertor.java file.
>
>     private static final String DATASET_TITLE = "Add DataSet.title here";
>     private static final String DATA_SOURCE_NAME = "Add DataSource.name here";
>
> I guess DATA_SOURCE_NAME = "test"
> if I had defined <property name="source.db.name" value="test"/> in XML. Am I right ?
> How about DATASET_TITLE? What values I am supposed to add for this variable ?
>
> -Bala
>
> ________________________________________
> From: Richard Smith [[hidden email]]
> Sent: Monday, June 07, 2010 5:16 AM
> To: Selvaraj, Balamuruganand
> Cc: [hidden email]; [hidden email]
> Subject: Re: Custom DB incorporation in Intermine
>
> Hi Bala,
>
> Selvaraj, Balamuruganand wrote:
>> Hi Julie / Richard,
>>
>> I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
>>
>> 1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
>> Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?
>
> Is this a local database other than chado?  If reading from a chado
> database then it's best to build on the existing chado converter and
> processors.
>
> If you're reading from a different database you can use the make_source
> script to create a new 'db' source:
>
> bio/scripts/make_source test db
>
> Which will create a bio/sources/test directory including a TestConverter
> which extends BioDBConverter.  All of the parsers extend a DataConverter
> class with specific extensions for reading files, directories, DBs or
> Items XML.
>
> Parsers are called by including the source in project.xml.  In this case
> you would add 'test' with an attribute to identifier the database:
>
>    <property name="source.db.name" value="test-db"/>
>
> You set up database access properties for test-db in your
> .intermine/xxmine.properties file, just as for chado database:
>
> db.test.datasource.serverName=servername
> db.test.datasource.databaseName=database-name
> db.test.datasource.user=user
> db.test.datasource.password=password
>
> db.test.datasource.class=org.postgresql.jdbc3.Jdbc3PoolingDataSource
> db.test.datasource.dataSourceName=db.test
> db.test.datasource.maxConnections=10
> db.test.driver=org.postgresql.Driver
> db.test.platform=PostgreSQL
>
>
> So the TestConverter can now be called as with other sources by running
>  > ant -Dsource=test from the xxmine/integrate.  It has a Database object
> and connection already set up.
>
> You can read from the source database and create and store Items just
> as with other types of source.  See an example, this reads from a file
> but the item creation and store() method works exactly the same:
>
> http://intrac.flymine.org/browser/branches/intermine_0_93/bio/sources/wormbase-identifiers/main/src/org/intermine/bio/dataconversion/WormBaseIdentifiersConverter.java
>
>
>
>> 2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.
>
>
> Do you mean if one object you store references another object?  There is
> some information about how the mapping to the database works here:
>
> http://www.intermine.org/wiki/GettingStarted#a6.2Creatingadatabase
>
> For references to a single object there usually a foreign key in the
> first table.  e.g. Gene has a reference called 'organism' to the
> Organism class.  In the database the gene table has an organismid
> column.
>
> For collections there will be an additional indirection table created,
> e.g. Gene has a collection of Protein objects, there will be an extra
> genesproteins table created.
>
>
> I hope this helps, please let me know if you have any further questions.
>
> Regards,
> Richard.
>
>
>
>
>
>> Thanks for your reply.
>>
>> Best regards,
>> Bala
>>
>>

_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev

flybase_intermine.log (10K) Download Attachment
project.properties (880 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Custom DB incorporation in Intermine

Richard Smith
In reply to this post by Selvaraj, Balamuruganand
Hi Bala,
Most objects in InterMine databases have a collection of DataSets.  Each
source provides a DataSet object that contains some description of what
is being loaded.  The DataSet references a DataSource which shows where
the data came from.

So for example, data from UniProt in FlyMine has

DataSet.title = Swiss-Prot data set
DataSource.name = UniProt

So you can just put descriptive text in these variables.


The db.name property in the project.xml isn't related to these.  I think
the project.xml entry should look like:

     <source name="test" type="test">
       <property name="source.db.name" value = "test"/>
     </source>


Cheers,
Richard.




Selvaraj, Balamuruganand wrote:

> Richard,
>
> Thanks for your reply.
>
> The testConvertor was created without any issues. Could you tell me what I might have to add for these class level variables defined in the  testConvertor.java file.
>
>     private static final String DATASET_TITLE = "Add DataSet.title here";
>     private static final String DATA_SOURCE_NAME = "Add DataSource.name here";
>
> I guess DATA_SOURCE_NAME = "test"
> if I had defined <property name="source.db.name" value="test"/> in XML. Am I right ?
> How about DATASET_TITLE? What values I am supposed to add for this variable ?
>
> -Bala
>
> ________________________________________
> From: Richard Smith [[hidden email]]
> Sent: Monday, June 07, 2010 5:16 AM
> To: Selvaraj, Balamuruganand
> Cc: [hidden email]; [hidden email]
> Subject: Re: Custom DB incorporation in Intermine
>
> Hi Bala,
>
> Selvaraj, Balamuruganand wrote:
>> Hi Julie / Richard,
>>
>> I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
>>
>> 1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
>> Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?
>
> Is this a local database other than chado?  If reading from a chado
> database then it's best to build on the existing chado converter and
> processors.
>
> If you're reading from a different database you can use the make_source
> script to create a new 'db' source:
>
> bio/scripts/make_source test db
>
> Which will create a bio/sources/test directory including a TestConverter
> which extends BioDBConverter.  All of the parsers extend a DataConverter
> class with specific extensions for reading files, directories, DBs or
> Items XML.
>
> Parsers are called by including the source in project.xml.  In this case
> you would add 'test' with an attribute to identifier the database:
>
>    <property name="source.db.name" value="test-db"/>
>
> You set up database access properties for test-db in your
> .intermine/xxmine.properties file, just as for chado database:
>
> db.test.datasource.serverName=servername
> db.test.datasource.databaseName=database-name
> db.test.datasource.user=user
> db.test.datasource.password=password
>
> db.test.datasource.class=org.postgresql.jdbc3.Jdbc3PoolingDataSource
> db.test.datasource.dataSourceName=db.test
> db.test.datasource.maxConnections=10
> db.test.driver=org.postgresql.Driver
> db.test.platform=PostgreSQL
>
>
> So the TestConverter can now be called as with other sources by running
>  > ant -Dsource=test from the xxmine/integrate.  It has a Database object
> and connection already set up.
>
> You can read from the source database and create and store Items just
> as with other types of source.  See an example, this reads from a file
> but the item creation and store() method works exactly the same:
>
> http://intrac.flymine.org/browser/branches/intermine_0_93/bio/sources/wormbase-identifiers/main/src/org/intermine/bio/dataconversion/WormBaseIdentifiersConverter.java
>
>
>
>> 2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.
>
>
> Do you mean if one object you store references another object?  There is
> some information about how the mapping to the database works here:
>
> http://www.intermine.org/wiki/GettingStarted#a6.2Creatingadatabase
>
> For references to a single object there usually a foreign key in the
> first table.  e.g. Gene has a reference called 'organism' to the
> Organism class.  In the database the gene table has an organismid
> column.
>
> For collections there will be an additional indirection table created,
> e.g. Gene has a collection of Protein objects, there will be an extra
> genesproteins table created.
>
>
> I hope this helps, please let me know if you have any further questions.
>
> Regards,
> Richard.
>
>
>
>
>
>> Thanks for your reply.
>>
>> Best regards,
>> Bala
>>
>>


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Custom DB incorporation in Intermine

Richard Smith
In reply to this post by Selvaraj, Balamuruganand
The problem could be with any of the project.properties files in the
dependency chain - i.e. any project mentioned in a compile.dependencies
property.  And it seems as though one of them has compile.dependencies
or extra.project.dependencies property that is empty.

The error message isn't very good so though so it doesn't give an
indication which one.  Try having a look through flybase/dbmodel,
the flybase source, etc.

If you can't find it can you send the entire output from running the
command, i.e. capture all the terminal output.

Thanks,
Richard.




Selvaraj, Balamuruganand wrote:

> Richard,
>
> Please find attached the Failed Build message that I got while running integrate. (Attachment as : flybase_intermine.log)
>
> Yes, the project.properties seems to having all dependencies. But still I get this Null pointer error.
> I have also attached the project.properties just for your reference.
>
> Best regards,
> Bala
> ________________________________________
> From: Richard Smith [[hidden email]]
> Sent: Monday, June 07, 2010 3:40 PM
> To: Selvaraj, Balamuruganand
> Cc: [hidden email]; [hidden email]; Goodman, Joshua Lawrence
> Subject: Re: Custom DB incorporation in Intermine
>
> Hi Bala,
> Could you run the same command with the -v flag to get verbose output
> and send the whole stack trace please?
>
>
> If you created the flybase Mine with the make_mine script the properties
> should have been set up correctly by default.  The flymine/integrate
> project.properties look like this (minus commnets):
>
>
> compile.dependencies = intermine/integrate/main,\
>                         flymine/dbmodel,\
>                         bio/core/main
>
>
> intermine.properties.file = flymine.properties
> default.intermine.properties.file =
> ../default.intermine.integrate.properties
>
> extra.project.dependencies = flymine/dbmodel
>
>
>
>
> Thanks,
> Richard.
>
>
>
> Selvaraj, Balamuruganand wrote:
>> Richard,
>>
>> I have another question. When I run from the intergrate directory ant -Dsource=flybase-db, I get a Null pointer error as given below
>>
>> java.lang.NullPointerException
>>   at org.intermine.task.Dependencies.followProjectDependencies(Dependencies.java:455)
>>
>> However, I have defined the project dependencies in flybase/intergrate/project properties as
>> extra.project.dependencies = flybase/dbmodel
>>
>> I am not sure as to why I am getting a Null pointer error. Can you suggest what I am doing wrong ?
>>
>> Thanks.
>>
>> Best regards,
>> Bala
>> ________________________________________
>> From: Selvaraj, Balamuruganand
>> Sent: Monday, June 07, 2010 12:29 PM
>> To: Richard Smith
>> Cc: [hidden email]; [hidden email]; Goodman, Joshua Lawrence
>> Subject: RE: Custom DB incorporation in Intermine
>>
>> Richard,
>>
>> Thanks for your reply.
>>
>> The testConvertor was created without any issues. Could you tell me what I might have to add for these class level variables defined in the  testConvertor.java file.
>>
>>     private static final String DATASET_TITLE = "Add DataSet.title here";
>>     private static final String DATA_SOURCE_NAME = "Add DataSource.name here";
>>
>> I guess DATA_SOURCE_NAME = "test"
>> if I had defined <property name="source.db.name" value="test"/> in XML. Am I right ?
>> How about DATASET_TITLE? What values I am supposed to add for this variable ?
>>
>> -Bala
>>
>> ________________________________________
>> From: Richard Smith [[hidden email]]
>> Sent: Monday, June 07, 2010 5:16 AM
>> To: Selvaraj, Balamuruganand
>> Cc: [hidden email]; [hidden email]
>> Subject: Re: Custom DB incorporation in Intermine
>>
>> Hi Bala,
>>
>> Selvaraj, Balamuruganand wrote:
>>> Hi Julie / Richard,
>>>
>>> I have few questions/doubts as I am trying to learn & understand ways to incorporate datas from local custom DB into the intermine system. I have the following questions in regard to that
>>>
>>> 1.  I have made the DBmodels for my database. Relevant tables are also properly getting created in the DB. I am now in the process of writing the custom Processor. I am not really sure as to how the Processors are being called.
>>> Is there any super class for all processor. Could you let me know as to how the processors are being called once they are declared in the project.xml. Is there any specific method which need to be incorporated ?
>> Is this a local database other than chado?  If reading from a chado
>> database then it's best to build on the existing chado converter and
>> processors.
>>
>> If you're reading from a different database you can use the make_source
>> script to create a new 'db' source:
>>
>> bio/scripts/make_source test db
>>
>> Which will create a bio/sources/test directory including a TestConverter
>> which extends BioDBConverter.  All of the parsers extend a DataConverter
>> class with specific extensions for reading files, directories, DBs or
>> Items XML.
>>
>> Parsers are called by including the source in project.xml.  In this case
>> you would add 'test' with an attribute to identifier the database:
>>
>>    <property name="source.db.name" value="test-db"/>
>>
>> You set up database access properties for test-db in your
>> .intermine/xxmine.properties file, just as for chado database:
>>
>> db.test.datasource.serverName=servername
>> db.test.datasource.databaseName=database-name
>> db.test.datasource.user=user
>> db.test.datasource.password=password
>>
>> db.test.datasource.class=org.postgresql.jdbc3.Jdbc3PoolingDataSource
>> db.test.datasource.dataSourceName=db.test
>> db.test.datasource.maxConnections=10
>> db.test.driver=org.postgresql.Driver
>> db.test.platform=PostgreSQL
>>
>>
>> So the TestConverter can now be called as with other sources by running
>>  > ant -Dsource=test from the xxmine/integrate.  It has a Database object
>> and connection already set up.
>>
>> You can read from the source database and create and store Items just
>> as with other types of source.  See an example, this reads from a file
>> but the item creation and store() method works exactly the same:
>>
>> http://intrac.flymine.org/browser/branches/intermine_0_93/bio/sources/wormbase-identifiers/main/src/org/intermine/bio/dataconversion/WormBaseIdentifiersConverter.java
>>
>>
>>
>>> 2. It would be great if you could tell me how the Class reference are being stored in the DB using the Processor i.e I want to know how objects are manipulated for storing obj reference in Java and then stored in DB.
>>
>> Do you mean if one object you store references another object?  There is
>> some information about how the mapping to the database works here:
>>
>> http://www.intermine.org/wiki/GettingStarted#a6.2Creatingadatabase
>>
>> For references to a single object there usually a foreign key in the
>> first table.  e.g. Gene has a reference called 'organism' to the
>> Organism class.  In the database the gene table has an organismid
>> column.
>>
>> For collections there will be an additional indirection table created,
>> e.g. Gene has a collection of Protein objects, there will be an extra
>> genesproteins table created.
>>
>>
>> I hope this helps, please let me know if you have any further questions.
>>
>> Regards,
>> Richard.
>>
>>
>>
>>
>>
>>> Thanks for your reply.
>>>
>>> Best regards,
>>> Bala
>>>
>>>
>


_______________________________________________
dev mailing list
[hidden email]
http://mail.intermine.org/cgi-bin/mailman/listinfo/dev