Command Based marteditor

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Command Based marteditor

ahmad.hassan

Hi,

I am working at Medical Research Council, Harwell, UK. We provide the
biomart interface to Europhenome database. We are trying to automate the
process of re-building biomart with new data. At the moment we need to run
marteditor manually and update the dataset in previously created mart. Is
there a command based marteditor tool available that allows us to script
everything for updating mart with the new data?

Thanks.

Europhenome Team
MRC, Harwell



This email may have a PROTECTIVE MARKING, for an explanation please see:
http://www.mrc.ac.uk/About/Informationandstandards/Documentmarking/index.htm
Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Joachim Baran
Hi!

  You might be interested in MartScript,
http://github.com/joejimbo/MartScript, which can be used to
automatically create Ensembl (and related) marts.

  I do not know your database structure, but perhaps it would work for
you with some minor modifications as well. The major downside of using
MartScript right now is obvious: I have been too lazy to document it
properly yet. If you have a go at it, please let me know and perhaps
we can figure out to get it working together. Are you around at Genome
Informatics next week? We could have a chat about it too.

Best Wishes,
Joachim

--
B.1079 Michael Smith Building
Faculty of Life Sciences
The University of Manchester
Oxford Road
Manchester
M13 9PT
United Kingdom
Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

ahmad.hassan

Hi,

Everytime we have a new data from medical clinics, we do the following
steps from MartEditor GUI in order to update biomart:

 File => Naive
 File => import
 File => update all
 File => export

Just because of this manual step, we cannot script/automate our whole data
upload process. It will be great if we can script the above functionality.
Do you reckon that MartScript will help us?

I am going to IMGC, Crete, Greece in October. Please let me know if you
will be there then we can discuss this further. Thanks.

Best Regards, Hassan
http://cern.ch/ahmadh/portfolio



> Hi!
>
>   You might be interested in MartScript,
> http://github.com/joejimbo/MartScript, which can be used to
> automatically create Ensembl (and related) marts.
>
>   I do not know your database structure, but perhaps it would work for
> you with some minor modifications as well. The major downside of using
> MartScript right now is obvious: I have been too lazy to document it
> properly yet. If you have a go at it, please let me know and perhaps
> we can figure out to get it working together. Are you around at Genome
> Informatics next week? We could have a chat about it too.
>
> Best Wishes,
> Joachim
>
> --
> B.1079 Michael Smith Building
> Faculty of Life Sciences
> The University of Manchester
> Oxford Road
> Manchester
> M13 9PT
> United Kingdom
>



This email may have a PROTECTIVE MARKING, for an explanation please see:
http://www.mrc.ac.uk/About/Informationandstandards/Documentmarking/index.htm
Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Joachim Baran
Hi,

On 8 September 2010 11:51,  <[hidden email]> wrote:
>  File => Naive
>  File => import
>  File => update all
>  File => export
> Do you reckon that MartScript will help us?
  'Naive' is not supported as such, but you could have a look at the
Java code and implement it yourself. It is fairly straightforward,
because I just call MartEditor code from within MartScript when
mimicking MartBuilder/MartEditor functionality. The other steps would
work though.

> I am going to IMGC, Crete, Greece in October. Please let me know if you
> will be there then we can discuss this further. Thanks.
  Unfortunately, I am not going there.

Joachim

--
B.1079 Michael Smith Building
Faculty of Life Sciences
The University of Manchester
Oxford Road
Manchester
M13 9PT
United Kingdom
Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Richard Holland
Word of warning - MartEditor will (highly likely) be replaced in BioMart 0.8, with entirely new source code producing an entirely new mart XML format.

cheers,
Richard

On 8 Sep 2010, at 12:05, Joachim Baran wrote:

> Hi,
>
> On 8 September 2010 11:51,  <[hidden email]> wrote:
>>  File => Naive
>>  File => import
>>  File => update all
>>  File => export
>> Do you reckon that MartScript will help us?
>  'Naive' is not supported as such, but you could have a look at the
> Java code and implement it yourself. It is fairly straightforward,
> because I just call MartEditor code from within MartScript when
> mimicking MartBuilder/MartEditor functionality. The other steps would
> work though.
>
>> I am going to IMGC, Crete, Greece in October. Please let me know if you
>> will be there then we can discuss this further. Thanks.
>  Unfortunately, I am not going there.
>
> Joachim
>
> --
> B.1079 Michael Smith Building
> Faculty of Life Sciences
> The University of Manchester
> Oxford Road
> Manchester
> M13 9PT
> United Kingdom

--
Richard Holland, BSc MBCS
Operations and Delivery Director, Eagle Genomics Ltd
T: +44 (0)1223 654481 ext 3 | E: [hidden email]
http://www.eaglegenomics.com/

Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Darren Oakley
Hi,

Just out of interest - is your data changing that rapidly between releases that you need to reconfigure/rebuild the MartEditor XMLs each build?

We automatically rebuild our biomarts nightly,  all we do is just re-run the SQL generated by MartBuilder (incorporating your new data) and reconfigure/restart your webserver.*

If you want some more info i'd be happy to show you some of our build code.

Thanks,

Daz

* This is a slightly simplified version.  As our mart build takes a few hours, (and we don't want our mart to be offline for that long), we run the MartBuilder SQL into a slave/redundant database and then when we reconfigure/restart the webserver we point it at the new database, so we get the minimum amount of downtime we can running off a single apache instance.

On 8 Sep 2010, at 12:14, Richard Holland wrote:

> Word of warning - MartEditor will (highly likely) be replaced in BioMart 0.8, with entirely new source code producing an entirely new mart XML format.
>
> cheers,
> Richard
>
> On 8 Sep 2010, at 12:05, Joachim Baran wrote:
>
>> Hi,
>>
>> On 8 September 2010 11:51,  <[hidden email]> wrote:
>>> File => Naive
>>> File => import
>>> File => update all
>>> File => export
>>> Do you reckon that MartScript will help us?
>> 'Naive' is not supported as such, but you could have a look at the
>> Java code and implement it yourself. It is fairly straightforward,
>> because I just call MartEditor code from within MartScript when
>> mimicking MartBuilder/MartEditor functionality. The other steps would
>> work though.
>>
>>> I am going to IMGC, Crete, Greece in October. Please let me know if you
>>> will be there then we can discuss this further. Thanks.
>> Unfortunately, I am not going there.
>>
>> Joachim
>>
>> --
>> B.1079 Michael Smith Building
>> Faculty of Life Sciences
>> The University of Manchester
>> Oxford Road
>> Manchester
>> M13 9PT
>> United Kingdom
>
> --
> Richard Holland, BSc MBCS
> Operations and Delivery Director, Eagle Genomics Ltd
> T: +44 (0)1223 654481 ext 3 | E: [hidden email]
> http://www.eaglegenomics.com/
>



--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Andrew Blake
  Hiya Darren
It's not really the data build that's the problem it's when we use the
actual data in the build to auto-populate a drop down in the filters
section of the biomart. When we build the next one the only way we've
found of resetting those filter dropdowns is to recreate them by hand
using the gui rather than scripted - If you don't have any drop downs
populated from the data you'll probably never run into this problem but
in case we are doing something daft any advice is welcome :-) We may
have to just lose the drop downs ??
Cheers
Andy

On 08/09/2010 13:52, Darren Oakley wrote:

> Hi,
>
> Just out of interest - is your data changing that rapidly between releases that you need to reconfigure/rebuild the MartEditor XMLs each build?
>
> We automatically rebuild our biomarts nightly,  all we do is just re-run the SQL generated by MartBuilder (incorporating your new data) and reconfigure/restart your webserver.*
>
> If you want some more info i'd be happy to show you some of our build code.
>
> Thanks,
>
> Daz
>
> * This is a slightly simplified version.  As our mart build takes a few hours, (and we don't want our mart to be offline for that long), we run the MartBuilder SQL into a slave/redundant database and then when we reconfigure/restart the webserver we point it at the new database, so we get the minimum amount of downtime we can running off a single apache instance.
>
> On 8 Sep 2010, at 12:14, Richard Holland wrote:
>
>> Word of warning - MartEditor will (highly likely) be replaced in BioMart 0.8, with entirely new source code producing an entirely new mart XML format.
>>
>> cheers,
>> Richard
>>
>> On 8 Sep 2010, at 12:05, Joachim Baran wrote:
>>
>>> Hi,
>>>
>>> On 8 September 2010 11:51,<[hidden email]>  wrote:
>>>> File =>  Naive
>>>> File =>  import
>>>> File =>  update all
>>>> File =>  export
>>>> Do you reckon that MartScript will help us?
>>> 'Naive' is not supported as such, but you could have a look at the
>>> Java code and implement it yourself. It is fairly straightforward,
>>> because I just call MartEditor code from within MartScript when
>>> mimicking MartBuilder/MartEditor functionality. The other steps would
>>> work though.
>>>
>>>> I am going to IMGC, Crete, Greece in October. Please let me know if you
>>>> will be there then we can discuss this further. Thanks.
>>> Unfortunately, I am not going there.
>>>
>>> Joachim
>>>
>>> --
>>> B.1079 Michael Smith Building
>>> Faculty of Life Sciences
>>> The University of Manchester
>>> Oxford Road
>>> Manchester
>>> M13 9PT
>>> United Kingdom
>> --
>> Richard Holland, BSc MBCS
>> Operations and Delivery Director, Eagle Genomics Ltd
>> T: +44 (0)1223 654481 ext 3 | E: [hidden email]
>> http://www.eaglegenomics.com/
>>
>
>

This email may have a PROTECTIVE MARKING, for an explanation please see:
http://www.mrc.ac.uk/About/Informationandstandards/Documentmarking/index.htm
Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Arek Kasprzyk-2
In reply to this post by Richard Holland
Re: [mart-dev] Command Based marteditor

Just a small comment here: So far to be released 0.8  has managed to capture the most of  the existing Meditor XML structures through the backwards compatibility

a



On 08/09/10 7:14 AM, "Richard Holland" <holland@...> wrote:

Word of warning - MartEditor will (highly likely) be replaced in BioMart 0.8, with entirely new source code producing an entirely new mart XML format.





cheers,
Richard

On 8 Sep 2010, at 12:05, Joachim Baran wrote:

> Hi,
>
> On 8 September 2010 11:51,  <a.hassan@...> wrote:
>>  File => Naive
>>  File => import
>>  File => update all
>>  File => export
>> Do you reckon that MartScript will help us?
>  'Naive' is not supported as such, but you could have a look at the
> Java code and implement it yourself. It is fairly straightforward,
> because I just call MartEditor code from within MartScript when
> mimicking MartBuilder/MartEditor functionality. The other steps would
> work though.
>
>> I am going to IMGC, Crete, Greece in October. Please let me know if you
>> will be there then we can discuss this further. Thanks.
>  Unfortunately, I am not going there.
>
> Joachim
>
> --
> B.1079 Michael Smith Building
> Faculty of Life Sciences
> The University of Manchester
> Oxford Road
> Manchester
> M13 9PT
> United Kingdom

--
Richard Holland, BSc MBCS
Operations and Delivery Director, Eagle Genomics Ltd
T: +44 (0)1223 654481 ext 3 | E: holland@...
http://www.eaglegenomics.com/






Arek Kasprzyk
Director, Bioinformatics Operations and Principal Investigator

Ontario Institute for Cancer Research
MaRS Centre, South Tower
101 College Street, Suite 800
Toronto, Ontario, Canada M5G 0A3
            
Tel:                        416-673-8559
Toll-free:             1-866-678-6427
www.oicr.on.ca
 
This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
 


Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Arek Kasprzyk-2
In reply to this post by ahmad.hassan
Re: [mart-dev] Command Based marteditor
Hi Hassan,
There has been a number of similar requests in the past. We are planning to properly expose the update API in the 0.8.


a



On 08/09/10 6:51 AM, "a.hassan@..." <a.hassan@...> wrote:



Hi,

Everytime we have a new data from medical clinics, we do the following
steps from MartEditor GUI in order to update biomart:

 File => Naive
 File => import
 File => update all
 File => export

Just because of this manual step, we cannot script/automate our whole data
upload process. It will be great if we can script the above functionality.
Do you reckon that MartScript will help us?

I am going to IMGC, Crete, Greece in October. Please let me know if you
will be there then we can discuss this further. Thanks.

Best Regards, Hassan
http://cern.ch/ahmadh/portfolio



> Hi!
>
>   You might be interested in MartScript,
> http://github.com/joejimbo/MartScript, which can be used to
> automatically create Ensembl (and related) marts.
>
>   I do not know your database structure, but perhaps it would work for
> you with some minor modifications as well. The major downside of using
> MartScript right now is obvious: I have been too lazy to document it
> properly yet. If you have a go at it, please let me know and perhaps
> we can figure out to get it working together. Are you around at Genome
> Informatics next week? We could have a chat about it too.
>
> Best Wishes,
> Joachim
>
> --
> B.1079 Michael Smith Building
> Faculty of Life Sciences
> The University of Manchester
> Oxford Road
> Manchester
> M13 9PT
> United Kingdom
>



This email may have a PROTECTIVE MARKING, for an explanation please see:
http://www.mrc.ac.uk/About/Informationandstandards/Documentmarking/index.htm





Arek Kasprzyk
Director, Bioinformatics Operations and Principal Investigator

Ontario Institute for Cancer Research
MaRS Centre, South Tower
101 College Street, Suite 800
Toronto, Ontario, Canada M5G 0A3
            
Tel:                        416-673-8559
Toll-free:             1-866-678-6427
www.oicr.on.ca
 
This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
 


Reply | Threaded
Open this post in threaded view
|

Re: Command Based marteditor

Darren Oakley
In reply to this post by Andrew Blake
Hi Andy,

Ahhh, no we don't have any really big drop down filters like yours...

Do you really need drop downs that big*, do you know if anyone's using them? (I personally find overly large drop downs a bit cumbersome).

For the ESLIM and MP names/id's, you could take an approach similar to how EnsEMBL handles Gene Ontology, (pretty much just text boxes, but these are controlled vocabularies so users can look up what they want to search by elsewhere):

http://www.biomart.org/biomart/martview?VIRTUALSCHEMANAME=default&ATTRIBUTES=mmusculus_gene_ensembl.default.feature_page.ensembl_gene_id|mmusculus_gene_ensembl.default.feature_page.ensembl_transcript_id&FILTERS=&VISIBLEPANEL=filterpanel

And for the Gene/Line names you could offer a text box that runs a LIKE query on the database, (also you need to make sure your db is doing case-insensitve queries at this point).

This would remove your need for the large drop downs and simplify your release process...

Sorry I couldn't be much more help.

Daz

* As a side note, last time I built the IKMC mart I got a bunch of warnings for your filters:

default ... Europhenome .............. 001/001 ... europhenome ............................... (WEB) www.europhenome.org:80 ........
Warning: Too many Options for filter [ line_name ] possible rendering problems for  Martview    
Warning: Too many Options for filter [ parameter_eslim_id ] possible rendering problems for  Martview    
Warning: Too many Options for filter [ parameter_name ] possible rendering problems for  Martview    
Warning: Too many Options for filter [ mp_term_name ] possible rendering problems for  Martview     OK

Do the drop downs contain all of your expected values?

On 8 Sep 2010, at 14:09, Andrew Blake wrote:

>
> Hiya Darren
> It's not really the data build that's the problem it's when we use the actual data in the build to auto-populate a drop down in the filters section of the biomart. When we build the next one the only way we've found of resetting those filter dropdowns is to recreate them by hand using the gui rather than scripted - If you don't have any drop downs populated from the data you'll probably never run into this problem but in case we are doing something daft any advice is welcome :-) We may have to just lose the drop downs ??
> Cheers
> Andy
>
> On 08/09/2010 13:52, Darren Oakley wrote:
>> Hi,
>>
>> Just out of interest - is your data changing that rapidly between releases that you need to reconfigure/rebuild the MartEditor XMLs each build?
>>
>> We automatically rebuild our biomarts nightly,  all we do is just re-run the SQL generated by MartBuilder (incorporating your new data) and reconfigure/restart your webserver.*
>>
>> If you want some more info i'd be happy to show you some of our build code.
>>
>> Thanks,
>>
>> Daz
>>
>> * This is a slightly simplified version.  As our mart build takes a few hours, (and we don't want our mart to be offline for that long), we run the MartBuilder SQL into a slave/redundant database and then when we reconfigure/restart the webserver we point it at the new database, so we get the minimum amount of downtime we can running off a single apache instance.
>>
>> On 8 Sep 2010, at 12:14, Richard Holland wrote:
>>
>>> Word of warning - MartEditor will (highly likely) be replaced in BioMart 0.8, with entirely new source code producing an entirely new mart XML format.
>>>
>>> cheers,
>>> Richard
>>>
>>> On 8 Sep 2010, at 12:05, Joachim Baran wrote:
>>>
>>>> Hi,
>>>>
>>>> On 8 September 2010 11:51,<[hidden email]>  wrote:
>>>>> File =>  Naive
>>>>> File =>  import
>>>>> File =>  update all
>>>>> File =>  export
>>>>> Do you reckon that MartScript will help us?
>>>> 'Naive' is not supported as such, but you could have a look at the
>>>> Java code and implement it yourself. It is fairly straightforward,
>>>> because I just call MartEditor code from within MartScript when
>>>> mimicking MartBuilder/MartEditor functionality. The other steps would
>>>> work though.
>>>>
>>>>> I am going to IMGC, Crete, Greece in October. Please let me know if you
>>>>> will be there then we can discuss this further. Thanks.
>>>> Unfortunately, I am not going there.
>>>>
>>>> Joachim
>>>>
>>>> --
>>>> B.1079 Michael Smith Building
>>>> Faculty of Life Sciences
>>>> The University of Manchester
>>>> Oxford Road
>>>> Manchester
>>>> M13 9PT
>>>> United Kingdom
>>> --
>>> Richard Holland, BSc MBCS
>>> Operations and Delivery Director, Eagle Genomics Ltd
>>> T: +44 (0)1223 654481 ext 3 | E: [hidden email]
>>> http://www.eaglegenomics.com/
>>>
>>
>>
>
> This email may have a PROTECTIVE MARKING, for an explanation please see:
> http://www.mrc.ac.uk/About/Informationandstandards/Documentmarking/index.htm



--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.