transfer entire track to user-created annotations

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

transfer entire track to user-created annotations

Brian.Mack

Hi,

 

I’ve successfully loaded my Maker gff3 into WebApollo on the provided virtual machine. I’m unclear about the typical workflow for saving your edited genes with the rest of the unedited annotations. If I have a set of genes that I have modified in WebApollo, should I just save them as a gff3 and then somehow merge them with rest of the genes? Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous thread (http://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.

 

Thanks,

Brian





This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn


> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>
> Hi,
>
>  
>
> I’ve successfully loaded my Maker gff3 into WebApollo on the provided virtual machine. I’m unclear about the typical workflow for saving your edited genes with the rest of the unedited annotations. If I have a set of genes that I have modified
>

I think that that is what is most often done.  

> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>
>

In theory this should be in the last few releases based on this:  http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))

I think you use the script under tools:  

add_transcripts_from_gff3_to_annotations.pl

Nathan

> Thanks,
>
> Brian
>
>
>
>
>
> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.


Reply | Threaded
Open this post in threaded view
|

RE: transfer entire track to user-created annotations

Brian.Mack
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:

[1] 9422
[2] 9423
webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.

Am I using the correct URL format for this script?

Brian



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Tuesday, October 21, 2014 3:36 PM
To: [hidden email]
Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
Subject: Re: [apollo] transfer entire track to user-created annotations



> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>
> Hi,
>
>  
>
> I’ve successfully loaded my Maker gff3 into WebApollo on the provided virtual machine. I’m unclear about the typical workflow for saving your edited genes with the rest of the unedited annotations. If I have a set of genes that I have modified
>

I think that that is what is most often done.  

> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>
>

In theory this should be in the last few releases based on this:  http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))

I think you use the script under tools:  

add_transcripts_from_gff3_to_annotations.pl

Nathan

> Thanks,
>
> Brian
>
>
>
>
>
> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.



Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn
Do you have the full command?

If you are running it locally I would imagine it would be something like:


./tools/data/add_transcripts_from_gff3_to_annotations.pl -U localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin
-i path/to/my.gff

I’m wondering if you didn’t supply a user-name with -u?

Nathan

> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>
> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>
> [1] 9422
> [2] 9423
> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>
> Am I using the correct URL format for this script?
>
> Brian
>
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Tuesday, October 21, 2014 3:36 PM
> To: [hidden email]
> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
>
>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>
>> Hi,
>>
>>
>>
>> I’ve successfully loaded my Maker gff3 into WebApollo on the provided virtual machine. I’m unclear about the typical workflow for saving your edited genes with the rest of the unedited annotations. If I have a set of genes that I have modified
>>
>
> I think that that is what is most often done.  
>
>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>
>>
>
> In theory this should be in the last few releases based on this:  http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>
> I think you use the script under tools:  
>
> add_transcripts_from_gff3_to_annotations.pl
>
> Nathan
>
>> Thanks,
>>
>> Brian
>>
>>
>>
>>
>>
>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>
>
>


Reply | Threaded
Open this post in threaded view
|

RE: transfer entire track to user-created annotations

Brian.Mack
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Thanks. I am running it locally. I'm using the following command:

add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p web_apollo_admin -U localhost:8080/WebApollo -i /home/webapollo/maker.gff

But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.

Brian


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Friday, October 24, 2014 12:06 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations

Do you have the full command?

If you are running it locally I would imagine it would be something like:


./tools/data/add_transcripts_from_gff3_to_annotations.pl -U localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin
-i path/to/my.gff

I’m wondering if you didn’t supply a user-name with -u?

Nathan

> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>
> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>
> [1] 9422
> [2] 9423
> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>
> Am I using the correct URL format for this script?
>
> Brian
>
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Tuesday, October 21, 2014 3:36 PM
> To: [hidden email]
> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
>
>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>
>> Hi,
>>
>>
>>
>> I’ve successfully loaded my Maker gff3 into WebApollo on the provided virtual machine. I’m unclear about the typical workflow for saving your edited genes with the rest of the unedited annotations. If I have a set of genes that I have modified
>>
>
> I think that that is what is most often done.  
>
>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>
>>
>
> In theory this should be in the last few releases based on this:  http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>
> I think you use the script under tools:  
>
> add_transcripts_from_gff3_to_annotations.pl
>
> Nathan
>
>> Thanks,
>>
>> Brian
>>
>>
>>
>>
>>
>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

How large is your maker.gff file?   Can you see your annotation file growing?  

I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  

Can you add print statements to that script and see where it hangs.  

Nathan

> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>
> Thanks. I am running it locally. I'm using the following command:
>
> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p web_apollo_admin -U localhost:8080/WebApollo -i /home/webapollo/maker.gff
>
> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>
> Brian
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Friday, October 24, 2014 12:06 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
> Do you have the full command?
>
> If you are running it locally I would imagine it would be something like:
>
>
> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin
> -i path/to/my.gff
>
> I’m wondering if you didn’t supply a user-name with -u?
>
> Nathan
>
>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>
>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>
>> [1] 9422
>> [2] 9423
>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>
>> Am I using the correct URL format for this script?
>>
>> Brian
>>
>>
>>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Tuesday, October 21, 2014 3:36 PM
>> To: [hidden email]
>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>> Subject: Re: [apollo] transfer entire track to user-created annotations
>>
>>
>>
>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>>
>>>
>>> I’ve successfully loaded my Maker gff3 into WebApollo on the provided virtual machine. I’m unclear about the typical workflow for saving your edited genes with the rest of the unedited annotations. If I have a set of genes that I have modified
>>>
>>
>> I think that that is what is most often done.  
>>
>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>
>>>
>>
>> In theory this should be in the last few releases based on this:  http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>
>> I think you use the script under tools:  
>>
>> add_transcripts_from_gff3_to_annotations.pl
>>
>> Nathan
>>
>>> Thanks,
>>>
>>> Brian
>>>
>>>
>>>
>>>
>>>
>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>
>>
>>
>
>
>


Reply | Threaded
Open this post in threaded view
|

RE: transfer entire track to user-created annotations

Brian.Mack
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.

Brian

19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
###
880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
###
24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Friday, October 24, 2014 1:26 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations


How large is your maker.gff file?   Can you see your annotation file growing?  

I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  

Can you add print statements to that script and see where it hangs.  

Nathan

> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>
> Thanks. I am running it locally. I'm using the following command:
>
> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
> web_apollo_admin -U localhost:8080/WebApollo -i
> /home/webapollo/maker.gff
>
> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>
> Brian
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Friday, October 24, 2014 12:06 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created
> annotations
>
> Do you have the full command?
>
> If you are running it locally I would imagine it would be something like:
>
>
> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
> path/to/my.gff
>
> I’m wondering if you didn’t supply a user-name with -u?
>
> Nathan
>
>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>
>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>
>> [1] 9422
>> [2] 9423
>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>
>> Am I using the correct URL format for this script?
>>
>> Brian
>>
>>
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Tuesday, October 21, 2014 3:36 PM
>> To: [hidden email]
>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>>
>>
>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>>
>>>
>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>> provided virtual machine. I’m unclear about the typical workflow for
>>> saving your edited genes with the rest of the unedited annotations.
>>> If I have a set of genes that I have modified
>>>
>>
>> I think that that is what is most often done.  
>>
>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>
>>>
>>
>> In theory this should be in the last few releases based on this:  
>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>
>> I think you use the script under tools:  
>>
>> add_transcripts_from_gff3_to_annotations.pl
>>
>> Nathan
>>
>>> Thanks,
>>>
>>> Brian
>>>
>>>
>>>
>>>
>>>
>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>
>>
>>
>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

I hate to ask you, but where does it get stuck to on convert_features?

Nathan

> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>
> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>
> Brian
>
> 19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
> 19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
> 19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
> 19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
> 19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> 19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> 19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
> 19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
> 19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
> 19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
> 19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> 19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> 19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
> 19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> ###
> 880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
> 880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
> 880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
> 880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
> 880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> 880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> ###
> 24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Friday, October 24, 2014 1:26 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> How large is your maker.gff file?   Can you see your annotation file growing?  
>
> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  
>
> Can you add print statements to that script and see where it hangs.  
>
> Nathan
>
>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>
>> Thanks. I am running it locally. I'm using the following command:
>>
>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>> web_apollo_admin -U localhost:8080/WebApollo -i
>> /home/webapollo/maker.gff
>>
>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>
>> Brian
>>
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Friday, October 24, 2014 12:06 PM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>> Do you have the full command?
>>
>> If you are running it locally I would imagine it would be something like:
>>
>>
>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>> path/to/my.gff
>>
>> I’m wondering if you didn’t supply a user-name with -u?
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>
>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>
>>> [1] 9422
>>> [2] 9423
>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>
>>> Am I using the correct URL format for this script?
>>>
>>> Brian
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>> To: [hidden email]
>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>> Subject: Re: [apollo] transfer entire track to user-created
>>> annotations
>>>
>>>
>>>
>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>> provided virtual machine. I’m unclear about the typical workflow for
>>>> saving your edited genes with the rest of the unedited annotations.
>>>> If I have a set of genes that I have modified
>>>>
>>>
>>> I think that that is what is most often done.  
>>>
>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>
>>>>
>>>
>>> In theory this should be in the last few releases based on this:  
>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>
>>> I think you use the script under tools:  
>>>
>>> add_transcripts_from_gff3_to_annotations.pl
>>>
>>> Nathan
>>>
>>>> Thanks,
>>>>
>>>> Brian
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>
>>>
>>>
>>
>>
>>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

I have a few more questions:

1 - does it always get stuck in the same place?  
2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?

Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.  

Nathan

> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>
>
> I hate to ask you, but where does it get stuck to on convert_features?
>
> Nathan
>
>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>
>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>
>> Brian
>>
>> 19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>> 19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>> 19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>> 19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>> 19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> ###
>> 880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>> 880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>> 880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> 880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> 880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> 880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> ###
>> 24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>
>>
>>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Friday, October 24, 2014 1:26 PM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created annotations
>>
>>
>> How large is your maker.gff file?   Can you see your annotation file growing?  
>>
>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  
>>
>> Can you add print statements to that script and see where it hangs.  
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>
>>> Thanks. I am running it locally. I'm using the following command:
>>>
>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>> /home/webapollo/maker.gff
>>>
>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>
>>> Brian
>>>
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>> Sent: Friday, October 24, 2014 12:06 PM
>>> To: [hidden email]
>>> Subject: Re: [apollo] transfer entire track to user-created
>>> annotations
>>>
>>> Do you have the full command?
>>>
>>> If you are running it locally I would imagine it would be something like:
>>>
>>>
>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>> path/to/my.gff
>>>
>>> I’m wondering if you didn’t supply a user-name with -u?
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>
>>>> [1] 9422
>>>> [2] 9423
>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>
>>>> Am I using the correct URL format for this script?
>>>>
>>>> Brian
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>> To: [hidden email]
>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>>
>>>>
>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>> provided virtual machine. I’m unclear about the typical workflow for
>>>>> saving your edited genes with the rest of the unedited annotations.
>>>>> If I have a set of genes that I have modified
>>>>>
>>>>
>>>> I think that that is what is most often done.  
>>>>
>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>
>>>>>
>>>>
>>>> In theory this should be in the last few releases based on this:  
>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>
>>>> I think you use the script under tools:  
>>>>
>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>
>>>> Nathan
>>>>
>>>>> Thanks,
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>


Reply | Threaded
Open this post in threaded view
|

RE: transfer entire track to user-created annotations

Brian.Mack
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Nathan,

I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).

Brian

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Saturday, October 25, 2014 11:41 AM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations


I have a few more questions:

1 - does it always get stuck in the same place?  
2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?

Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.  

Nathan

> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>
>
> I hate to ask you, but where does it get stuck to on convert_features?
>
> Nathan
>
>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>
>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>
>> Brian
>>
>> 19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>> 19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>> 19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>> 19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>> 19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>> 19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> 19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>> ###
>> 880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>> 880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>> 880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> 880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> 880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> 880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>> ###
>> 24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>
>>
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Friday, October 24, 2014 1:26 PM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>>
>> How large is your maker.gff file?   Can you see your annotation file growing?  
>>
>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  
>>
>> Can you add print statements to that script and see where it hangs.  
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>
>>> Thanks. I am running it locally. I'm using the following command:
>>>
>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>> /home/webapollo/maker.gff
>>>
>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>
>>> Brian
>>>
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>> Sent: Friday, October 24, 2014 12:06 PM
>>> To: [hidden email]
>>> Subject: Re: [apollo] transfer entire track to user-created
>>> annotations
>>>
>>> Do you have the full command?
>>>
>>> If you are running it locally I would imagine it would be something like:
>>>
>>>
>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>> path/to/my.gff
>>>
>>> I’m wondering if you didn’t supply a user-name with -u?
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>
>>>> [1] 9422
>>>> [2] 9423
>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>
>>>> Am I using the correct URL format for this script?
>>>>
>>>> Brian
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>> To: [hidden email]
>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>>
>>>>
>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>> If I have a set of genes that I have modified
>>>>>
>>>>
>>>> I think that that is what is most often done.  
>>>>
>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>
>>>>>
>>>>
>>>> In theory this should be in the last few releases based on this:  
>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>
>>>> I think you use the script under tools:  
>>>>
>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>
>>>> Nathan
>>>>
>>>>> Thanks,
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>



Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

Based on that, I’m wondering if you’re running out of memory within perl.  

That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.

So, last two things to try:
1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.  

If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.  

I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.


Nathan

> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
>
> --------
> WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
> --------
>
> Nathan,
>
> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
>
> Brian
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Saturday, October 25, 2014 11:41 AM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> I have a few more questions:
>
> 1 - does it always get stuck in the same place?  
> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
>
> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.  
>
> Nathan
>
>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>>
>>
>> I hate to ask you, but where does it get stuck to on convert_features?
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>>
>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>>
>>> Brian
>>>
>>> 19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>>> 19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>>> 19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>>> 19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>>> 19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> ###
>>> 880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>>> 880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>>> 880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> 880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> 880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> 880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> ###
>>> 24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>> Sent: Friday, October 24, 2014 1:26 PM
>>> To: [hidden email]
>>> Subject: Re: [apollo] transfer entire track to user-created
>>> annotations
>>>
>>>
>>> How large is your maker.gff file?   Can you see your annotation file growing?  
>>>
>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  
>>>
>>> Can you add print statements to that script and see where it hangs.  
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Thanks. I am running it locally. I'm using the following command:
>>>>
>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>>> /home/webapollo/maker.gff
>>>>
>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>>
>>>> Brian
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Friday, October 24, 2014 12:06 PM
>>>> To: [hidden email]
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>> Do you have the full command?
>>>>
>>>> If you are running it locally I would imagine it would be something like:
>>>>
>>>>
>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>>> path/to/my.gff
>>>>
>>>> I’m wondering if you didn’t supply a user-name with -u?
>>>>
>>>> Nathan
>>>>
>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>>
>>>>> [1] 9422
>>>>> [2] 9423
>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>>
>>>>> Am I using the correct URL format for this script?
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: [hidden email]
>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>>> To: [hidden email]
>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>> annotations
>>>>>
>>>>>
>>>>>
>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>>> If I have a set of genes that I have modified
>>>>>>
>>>>>
>>>>> I think that that is what is most often done.  
>>>>>
>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>>
>>>>>>
>>>>>
>>>>> In theory this should be in the last few releases based on this:  
>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>>
>>>>> I think you use the script under tools:  
>>>>>
>>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>>
>>>>> Nathan
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>


Reply | Threaded
Open this post in threaded view
|

RE: transfer entire track to user-created annotations

Brian.Mack
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.

So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Monday, October 27, 2014 12:24 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations


Based on that, I’m wondering if you’re running out of memory within perl.  

That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.

So, last two things to try:
1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.  

If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.  

I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.


Nathan

> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
>
> --------
> WARNING: At least one of the links in the message below goes to an IP
> address (e.g. 10.1.1.1), which could be malicious. To learn how to
> protect yourself, please go here: https://commons.lbl.gov/x/_591B
> --------
>
> Nathan,
>
> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
>
> Brian
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Saturday, October 25, 2014 11:41 AM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created
> annotations
>
>
> I have a few more questions:
>
> 1 - does it always get stuck in the same place?  
> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
>
> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.  
>
> Nathan
>
>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>>
>>
>> I hate to ask you, but where does it get stuck to on convert_features?
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>>
>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>>
>>> Brian
>>>
>>> 19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>>> 19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>>> 19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>> 19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>>> 19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>>> 19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> 19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>> ###
>>> 880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>>> 880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>>> 880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> 880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> 880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> 880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>> ###
>>> 24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>> Sent: Friday, October 24, 2014 1:26 PM
>>> To: [hidden email]
>>> Subject: Re: [apollo] transfer entire track to user-created
>>> annotations
>>>
>>>
>>> How large is your maker.gff file?   Can you see your annotation file growing?  
>>>
>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  
>>>
>>> Can you add print statements to that script and see where it hangs.  
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Thanks. I am running it locally. I'm using the following command:
>>>>
>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>>> /home/webapollo/maker.gff
>>>>
>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>>
>>>> Brian
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Friday, October 24, 2014 12:06 PM
>>>> To: [hidden email]
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>> Do you have the full command?
>>>>
>>>> If you are running it locally I would imagine it would be something like:
>>>>
>>>>
>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>>> path/to/my.gff
>>>>
>>>> I’m wondering if you didn’t supply a user-name with -u?
>>>>
>>>> Nathan
>>>>
>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>>
>>>>> [1] 9422
>>>>> [2] 9423
>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>>
>>>>> Am I using the correct URL format for this script?
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: [hidden email]
>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>>> To: [hidden email]
>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>> annotations
>>>>>
>>>>>
>>>>>
>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>>> If I have a set of genes that I have modified
>>>>>>
>>>>>
>>>>> I think that that is what is most often done.  
>>>>>
>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>>
>>>>>>
>>>>>
>>>>> In theory this should be in the last few releases based on this:  
>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>>
>>>>> I think you use the script under tools:  
>>>>>
>>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>>
>>>>> Nathan
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

> On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
>
> It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.

Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.

> So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?

;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.  

Colin said he’s going to take a look as well to double-check and may have an easier solution.

In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.

Nathan

>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Monday, October 27, 2014 12:24 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> Based on that, I’m wondering if you’re running out of memory within perl.  
>
> That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
>
> So, last two things to try:
> 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.  
>
> If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.  
>
> I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
>
>
> Nathan
>
>> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
>>
>> --------
>> WARNING: At least one of the links in the message below goes to an IP
>> address (e.g. 10.1.1.1), which could be malicious. To learn how to
>> protect yourself, please go here: https://commons.lbl.gov/x/_591B
>> --------
>>
>> Nathan,
>>
>> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
>>
>> Brian
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Saturday, October 25, 2014 11:41 AM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>>
>> I have a few more questions:
>>
>> 1 - does it always get stuck in the same place?  
>> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
>>
>> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.  
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>>>
>>>
>>> I hate to ask you, but where does it get stuck to on convert_features?
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>>>
>>>> Brian
>>>>
>>>> 19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>>>> 19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>>>> 19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>>>> 19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>>>> 19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> ###
>>>> 880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>>>> 880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>>>> 880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> ###
>>>> 24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Friday, October 24, 2014 1:26 PM
>>>> To: [hidden email]
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>>
>>>> How large is your maker.gff file?   Can you see your annotation file growing?  
>>>>
>>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  
>>>>
>>>> Can you add print statements to that script and see where it hangs.  
>>>>
>>>> Nathan
>>>>
>>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> Thanks. I am running it locally. I'm using the following command:
>>>>>
>>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>>>> /home/webapollo/maker.gff
>>>>>
>>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: [hidden email]
>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>> Sent: Friday, October 24, 2014 12:06 PM
>>>>> To: [hidden email]
>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>> annotations
>>>>>
>>>>> Do you have the full command?
>>>>>
>>>>> If you are running it locally I would imagine it would be something like:
>>>>>
>>>>>
>>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>>>> path/to/my.gff
>>>>>
>>>>> I’m wondering if you didn’t supply a user-name with -u?
>>>>>
>>>>> Nathan
>>>>>
>>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>
>>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>>>
>>>>>> [1] 9422
>>>>>> [2] 9423
>>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>>>
>>>>>> Am I using the correct URL format for this script?
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: [hidden email]
>>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>>>> To: [hidden email]
>>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>>> annotations
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>>>> If I have a set of genes that I have modified
>>>>>>>
>>>>>>
>>>>>> I think that that is what is most often done.  
>>>>>>
>>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> In theory this should be in the last few releases based on this:  
>>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>>>
>>>>>> I think you use the script under tools:  
>>>>>>
>>>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>


Reply | Threaded
Open this post in threaded view
|

RE: transfer entire track to user-created annotations

Brian.Mack
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Nathan,

I have all my maker predictions now copied to the user-created annotations, but for some reason WebApollo won't load the user-created annotations for the first 5 contigs. It just shows the spinning loading graphic. They are not much bigger than the other contigs that are loading fine. These contigs export to gff fine though. Any ideas?

Brian

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Monday, October 27, 2014 3:24 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations


> On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
>
> It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.

Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.

> So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?

;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.  

Colin said he’s going to take a look as well to double-check and may have an easier solution.

In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.

Nathan

>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Monday, October 27, 2014 12:24 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> Based on that, I’m wondering if you’re running out of memory within perl.  
>
> That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
>
> So, last two things to try:
> 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.  
>
> If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.  
>
> I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
>
>
> Nathan
>
>> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
>>
>> --------
>> WARNING: At least one of the links in the message below goes to an IP
>> address (e.g. 10.1.1.1), which could be malicious. To learn how to
>> protect yourself, please go here: https://commons.lbl.gov/x/_591B
>> --------
>>
>> Nathan,
>>
>> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
>>
>> Brian
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Saturday, October 25, 2014 11:41 AM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>>
>> I have a few more questions:
>>
>> 1 - does it always get stuck in the same place?  
>> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
>>
>> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.  
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>>>
>>>
>>> I hate to ask you, but where does it get stuck to on convert_features?
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>>>
>>>> Brian
>>>>
>>>> 19 maker gene 99910 100435 . + . ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>>>> 19 maker mRNA 99910 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>>>> 19 maker exon 99910 100048 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker exon 100329 100435 . + . ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker CDS 99910 100048 . + 0 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker CDS 100329 100435 . + 2 ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker gene 9939 10466 . + . ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>>>> 19 maker mRNA 9939 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>>>> 19 maker exon 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker exon 10124 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker five_prime_UTR 9939 9973 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker five_prime_UTR 10124 10131 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker CDS 10132 10299 . + 0 ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker three_prime_UTR 10300 10466 . + . ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> ###
>>>> 880 maker gene 1276 2894 . + . ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>>>> 880 maker mRNA 1276 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>>>> 880 maker exon 1276 1612 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880 maker exon 1843 2894 . + . ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880 maker CDS 1276 1612 . + 0 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880 maker CDS 1843 2894 . + 2 ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> ###
>>>> 24 maker gene 3911 5179 . + . ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Friday, October 24, 2014 1:26 PM
>>>> To: [hidden email]
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>>
>>>> How large is your maker.gff file?   Can you see your annotation file growing?  
>>>>
>>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?  
>>>>
>>>> Can you add print statements to that script and see where it hangs.  
>>>>
>>>> Nathan
>>>>
>>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> Thanks. I am running it locally. I'm using the following command:
>>>>>
>>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>>>> /home/webapollo/maker.gff
>>>>>
>>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: [hidden email]
>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>> Sent: Friday, October 24, 2014 12:06 PM
>>>>> To: [hidden email]
>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>> annotations
>>>>>
>>>>> Do you have the full command?
>>>>>
>>>>> If you are running it locally I would imagine it would be something like:
>>>>>
>>>>>
>>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>>>> path/to/my.gff
>>>>>
>>>>> I’m wondering if you didn’t supply a user-name with -u?
>>>>>
>>>>> Nathan
>>>>>
>>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>
>>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>>>
>>>>>> [1] 9422
>>>>>> [2] 9423
>>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>>>
>>>>>> Am I using the correct URL format for this script?
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: [hidden email]
>>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>>>> To: [hidden email]
>>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>>> annotations
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>>>> If I have a set of genes that I have modified
>>>>>>>
>>>>>>
>>>>>> I think that that is what is most often done.  
>>>>>>
>>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> In theory this should be in the last few releases based on this:  
>>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>>>
>>>>>> I think you use the script under tools:  
>>>>>>
>>>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

Colin
Hi Brian,
Would you be able to check your javascript console for any signs of errors?

-Colin

On Wed, Oct 29, 2014 at 10:54 AM, Mack, Brian <[hidden email]> wrote:
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Nathan,

I have all my maker predictions now copied to the user-created annotations, but for some reason WebApollo won't load the user-created annotations for the first 5 contigs. It just shows the spinning loading graphic. They are not much bigger than the other contigs that are loading fine. These contigs export to gff fine though. Any ideas?

Brian

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Monday, October 27, 2014 3:24 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations


> On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
>
> It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.

Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.

> So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?

;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.

Colin said he’s going to take a look as well to double-check and may have an easier solution.

In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.

Nathan

>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Monday, October 27, 2014 12:24 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> Based on that, I’m wondering if you’re running out of memory within perl.
>
> That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
>
> So, last two things to try:
> 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.
>
> If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.
>
> I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
>
>
> Nathan
>
>> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
>>
>> --------
>> WARNING: At least one of the links in the message below goes to an IP
>> address (e.g. 10.1.1.1), which could be malicious. To learn how to
>> protect yourself, please go here: https://commons.lbl.gov/x/_591B
>> --------
>>
>> Nathan,
>>
>> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
>>
>> Brian
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Saturday, October 25, 2014 11:41 AM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>>
>> I have a few more questions:
>>
>> 1 - does it always get stuck in the same place?
>> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
>>
>> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>>>
>>>
>>> I hate to ask you, but where does it get stuck to on convert_features?
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>>>
>>>> Brian
>>>>
>>>> 19 maker   gene    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>>>> 19 maker   mRNA    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>>>> 19 maker   exon    99910   100048  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   exon    100329  100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   CDS     99910   100048  .       +       0       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   CDS     100329  100435  .       +       2       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   gene    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>>>> 19 maker   mRNA    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>>>> 19 maker   exon    9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   exon    10124   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   five_prime_UTR  9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   five_prime_UTR  10124   10131   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   CDS     10132   10299   .       +       0       ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   three_prime_UTR 10300   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> ###
>>>> 880        maker   gene    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>>>> 880        maker   mRNA    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>>>> 880        maker   exon    1276    1612    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   exon    1843    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   CDS     1276    1612    .       +       0       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   CDS     1843    2894    .       +       2       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> ###
>>>> 24 maker   gene    3911    5179    .       +       .       ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Friday, October 24, 2014 1:26 PM
>>>> To: [hidden email]
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>>
>>>> How large is your maker.gff file?   Can you see your annotation file growing?
>>>>
>>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?
>>>>
>>>> Can you add print statements to that script and see where it hangs.
>>>>
>>>> Nathan
>>>>
>>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> Thanks. I am running it locally. I'm using the following command:
>>>>>
>>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>>>> /home/webapollo/maker.gff
>>>>>
>>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: [hidden email]
>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>> Sent: Friday, October 24, 2014 12:06 PM
>>>>> To: [hidden email]
>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>> annotations
>>>>>
>>>>> Do you have the full command?
>>>>>
>>>>> If you are running it locally I would imagine it would be something like:
>>>>>
>>>>>
>>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>>>> path/to/my.gff
>>>>>
>>>>> I’m wondering if you didn’t supply a user-name with -u?
>>>>>
>>>>> Nathan
>>>>>
>>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>
>>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>>>
>>>>>> [1] 9422
>>>>>> [2] 9423
>>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>>>
>>>>>> Am I using the correct URL format for this script?
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: [hidden email]
>>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>>>> To: [hidden email]
>>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>>> annotations
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>>>> If I have a set of genes that I have modified
>>>>>>>
>>>>>>
>>>>>> I think that that is what is most often done.
>>>>>>
>>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> In theory this should be in the last few releases based on this:
>>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>>>
>>>>>> I think you use the script under tools:
>>>>>>
>>>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>




Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

And if you check your tomcat logs (catalina.out) for the same that would be helpful.

Nathan

On Oct 29, 2014, at 9:21 AM, Colin <[hidden email]> wrote:

Hi Brian,
Would you be able to check your javascript console for any signs of errors?

-Colin

On Wed, Oct 29, 2014 at 10:54 AM, Mack, Brian <[hidden email]> wrote:
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Nathan,

I have all my maker predictions now copied to the user-created annotations, but for some reason WebApollo won't load the user-created annotations for the first 5 contigs. It just shows the spinning loading graphic. They are not much bigger than the other contigs that are loading fine. These contigs export to gff fine though. Any ideas?

Brian

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Monday, October 27, 2014 3:24 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations


> On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
>
> It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.

Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.

> So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?

;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.

Colin said he’s going to take a look as well to double-check and may have an easier solution.

In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.

Nathan

>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Monday, October 27, 2014 12:24 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> Based on that, I’m wondering if you’re running out of memory within perl.
>
> That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
>
> So, last two things to try:
> 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.
>
> If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.
>
> I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
>
>
> Nathan
>
>> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
>>
>> --------
>> WARNING: At least one of the links in the message below goes to an IP
>> address (e.g. 10.1.1.1), which could be malicious. To learn how to
>> protect yourself, please go here: https://commons.lbl.gov/x/_591B
>> --------
>>
>> Nathan,
>>
>> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
>>
>> Brian
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Saturday, October 25, 2014 11:41 AM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>>
>> I have a few more questions:
>>
>> 1 - does it always get stuck in the same place?
>> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
>>
>> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>>>
>>>
>>> I hate to ask you, but where does it get stuck to on convert_features?
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>>>
>>>> Brian
>>>>
>>>> 19 maker   gene    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>>>> 19 maker   mRNA    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>>>> 19 maker   exon    99910   100048  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   exon    100329  100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   CDS     99910   100048  .       +       0       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   CDS     100329  100435  .       +       2       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   gene    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>>>> 19 maker   mRNA    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>>>> 19 maker   exon    9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   exon    10124   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   five_prime_UTR  9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   five_prime_UTR  10124   10131   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   CDS     10132   10299   .       +       0       ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   three_prime_UTR 10300   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> ###
>>>> 880        maker   gene    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>>>> 880        maker   mRNA    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>>>> 880        maker   exon    1276    1612    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   exon    1843    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   CDS     1276    1612    .       +       0       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   CDS     1843    2894    .       +       2       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> ###
>>>> 24 maker   gene    3911    5179    .       +       .       ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Friday, October 24, 2014 1:26 PM
>>>> To: [hidden email]
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>>
>>>> How large is your maker.gff file?   Can you see your annotation file growing?
>>>>
>>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?
>>>>
>>>> Can you add print statements to that script and see where it hangs.
>>>>
>>>> Nathan
>>>>
>>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> Thanks. I am running it locally. I'm using the following command:
>>>>>
>>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>>>> /home/webapollo/maker.gff
>>>>>
>>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: [hidden email]
>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>> Sent: Friday, October 24, 2014 12:06 PM
>>>>> To: [hidden email]
>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>> annotations
>>>>>
>>>>> Do you have the full command?
>>>>>
>>>>> If you are running it locally I would imagine it would be something like:
>>>>>
>>>>>
>>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>>>> path/to/my.gff
>>>>>
>>>>> I’m wondering if you didn’t supply a user-name with -u?
>>>>>
>>>>> Nathan
>>>>>
>>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>
>>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>>>
>>>>>> [1] 9422
>>>>>> [2] 9423
>>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>>>
>>>>>> Am I using the correct URL format for this script?
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: [hidden email]
>>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>>>> To: [hidden email]
>>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>>> annotations
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>>>> If I have a set of genes that I have modified
>>>>>>>
>>>>>>
>>>>>> I think that that is what is most often done.
>>>>>>
>>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> In theory this should be in the last few releases based on this:
>>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>>>
>>>>>> I think you use the script under tools:
>>>>>>
>>>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>





Reply | Threaded
Open this post in threaded view
|

RE: transfer entire track to user-created annotations

Brian.Mack

--------
WARNING: At least one of the links in the message below goes to an IP address (e.g.10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Nathan, I copied my javascript console and tomcat logs below. There seems to be an error in the javascript console.

 

Brian

 

 

loaded WebApollo plugin main.js:36

Object

Browser.js:1528

In JBrowse configuration, datasets specified, but dataset_id not set. Dataset selector will not be shown. Browser.js:535

Annotation server error--maybe you forgot to login to the server? AnnotTrack.js:300

1.   TypeError

"TypeError: Cannot read property 'xhr' of undefined at http://10.114.138.235/WebApollo/jbrowse/plugins/WebApollo/js/View/Track/AnnotTrack.js:301:59 at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:37:21) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- rejected at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- Error at Promise.then.promise.then (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:252:24) at declare.setViewInfo (http://10.114.138.235/WebApollo/jbrowse/plugins/WebApollo/js/View/Track/AnnotTrack.js:255:16) at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2211:15) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 at http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2237:17 at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/Browser.js:1148:18) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 at runFactory (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1094:43) at execModule (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1223:5) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1258:6" instrumentation.js:18

1.   logErrorinstrumentation.js:18

'KeyboardEvent.keyLocation'' is deprecated. Please use 'KeyboardEvent.location' instead.

event.returnValue is deprecated. Please use the standard event.preventDefault() instead.

 


Oct 29, 2014 8:39:31 AM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-bio-8080"]

Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 339 ms

Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardService startInternal

INFO: Starting service Catalina

Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/7.0.26

Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/manager.xml

Oct 29, 2014 8:39:44 AM org.apache.catalina.util.SessionIdGenerator createSecureRandom

INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [12,606] milliseconds.

Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/host-manager.xml

Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT

Oct 29, 2014 8:39:45 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /var/lib/tomcat7/webapps/WebApollo

Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.ContextConfig validateSecurityRoles

INFO: WARNING: Security role name NobodyHasThisRole used in an <auth-constraint> without being defined in a <security-role>

Oct 29, 2014 8:39:46 AM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-bio-8080"]

Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.Catalina start

INFO: Server startup in 13530 ms

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
Sent: Wednesday, October 29, 2014 12:12 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations

 

 

And if you check your tomcat logs (catalina.out) for the same that would be helpful.

 

Nathan

 

On Oct 29, 2014, at 9:21 AM, Colin <[hidden email]> wrote:

 

Hi Brian,
Would you be able to check your javascript console for any signs of errors?

-Colin

 

On Wed, Oct 29, 2014 at 10:54 AM, Mack, Brian <[hidden email]> wrote:

--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Nathan,

I have all my maker predictions now copied to the user-created annotations, but for some reason WebApollo won't load the user-created annotations for the first 5 contigs. It just shows the spinning loading graphic. They are not much bigger than the other contigs that are loading fine. These contigs export to gff fine though. Any ideas?

Brian

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn

Sent: Monday, October 27, 2014 3:24 PM
To: [hidden email]
Subject: Re: [apollo] transfer entire track to user-created annotations


> On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
>
> It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.

Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.

> So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?

;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.

Colin said he’s going to take a look as well to double-check and may have an easier solution.

In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.

Nathan

>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Monday, October 27, 2014 12:24 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> Based on that, I’m wondering if you’re running out of memory within perl.
>
> That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
>
> So, last two things to try:
> 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.
>
> If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.
>
> I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
>
>
> Nathan
>
>> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
>>
>> --------
>> WARNING: At least one of the links in the message below goes to an IP
>> address (e.g. 10.1.1.1), which could be malicious. To learn how to
>> protect yourself, please go here: https://commons.lbl.gov/x/_591B
>> --------
>>
>> Nathan,
>>
>> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
>>
>> Brian
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>> Sent: Saturday, October 25, 2014 11:41 AM
>> To: [hidden email]
>> Subject: Re: [apollo] transfer entire track to user-created
>> annotations
>>
>>
>> I have a few more questions:
>>
>> 1 - does it always get stuck in the same place?
>> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
>>
>> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.
>>
>> Nathan
>>
>>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
>>>
>>>
>>> I hate to ask you, but where does it get stuck to on convert_features?
>>>
>>> Nathan
>>>
>>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
>>>>
>>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
>>>>
>>>> Brian
>>>>
>>>> 19 maker   gene    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
>>>> 19 maker   mRNA    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
>>>> 19 maker   exon    99910   100048  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   exon    100329  100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   CDS     99910   100048  .       +       0       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   CDS     100329  100435  .       +       2       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
>>>> 19 maker   gene    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
>>>> 19 maker   mRNA    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
>>>> 19 maker   exon    9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   exon    10124   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   five_prime_UTR  9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   five_prime_UTR  10124   10131   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   CDS     10132   10299   .       +       0       ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> 19 maker   three_prime_UTR 10300   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
>>>> ###
>>>> 880        maker   gene    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
>>>> 880        maker   mRNA    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
>>>> 880        maker   exon    1276    1612    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   exon    1843    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   CDS     1276    1612    .       +       0       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> 880        maker   CDS     1843    2894    .       +       2       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
>>>> ###
>>>> 24 maker   gene    3911    5179    .       +       .       ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>> Sent: Friday, October 24, 2014 1:26 PM
>>>> To: [hidden email]
>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>> annotations
>>>>
>>>>
>>>> How large is your maker.gff file?   Can you see your annotation file growing?
>>>>
>>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?
>>>>
>>>> Can you add print statements to that script and see where it hangs.
>>>>
>>>> Nathan
>>>>
>>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
>>>>>
>>>>> Thanks. I am running it locally. I'm using the following command:
>>>>>
>>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
>>>>> web_apollo_admin -U localhost:8080/WebApollo -i
>>>>> /home/webapollo/maker.gff
>>>>>
>>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: [hidden email]
>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>> Sent: Friday, October 24, 2014 12:06 PM
>>>>> To: [hidden email]
>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>> annotations
>>>>>
>>>>> Do you have the full command?
>>>>>
>>>>> If you are running it locally I would imagine it would be something like:
>>>>>
>>>>>
>>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
>>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
>>>>> path/to/my.gff
>>>>>
>>>>> I’m wondering if you didn’t supply a user-name with -u?
>>>>>
>>>>> Nathan
>>>>>
>>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>
>>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
>>>>>>
>>>>>> [1] 9422
>>>>>> [2] 9423
>>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
>>>>>>
>>>>>> Am I using the correct URL format for this script?
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: [hidden email]
>>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
>>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
>>>>>> To: [hidden email]
>>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
>>>>>> Subject: Re: [apollo] transfer entire track to user-created
>>>>>> annotations
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
>>>>>>> provided virtual machine. I’m unclear about the typical workflow
>>>>>>> for saving your edited genes with the rest of the unedited annotations.
>>>>>>> If I have a set of genes that I have modified
>>>>>>>
>>>>>>
>>>>>> I think that that is what is most often done.
>>>>>>
>>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> In theory this should be in the last few releases based on this:
>>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
>>>>>>
>>>>>> I think you use the script under tools:
>>>>>>
>>>>>> add_transcripts_from_gff3_to_annotations.pl
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>


 

 

Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

Hmm . . it is failing roughly here (not sure the version, but I doubt its changed much).

If it works for some tracks and not others, I’m wondering if annotations for the other track simply did not exist.

> xhr(context_path + "/AnnotationEditorService", {
>     handleAs: "json",
>     data: '{ "track": "' + track.getUniqueTrackName() + '", "operation": "get_features" }',
>     method: "post"
> }).then(function(response, ioArgs) {
>     var responseFeatures = response.features;
> ….
> , function(response, ioArgs) { //
>     console.log("Annotation server error--maybe you forgot to login to the server?");
>     track.handleError({ responseText: response.response.text } );
> // . . .
>     return response; //
> });

If you repeat the process (for the first 5) does it do the same?  

Nathan

> On Oct 29, 2014, at 11:35 AM, Mack, Brian <[hidden email]> wrote:
>
> --------
> WARNING: At least one of the links in the message below goes to an IP address (e.g.10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
> --------
>
> Nathan, I copied my javascript console and tomcat logs below. There seems to be an error in the javascript console.
>  
> Brian
>  
>  
> loaded WebApollo plugin main.js:36
> Object
> Browser.js:1528
> In JBrowse configuration, datasets specified, but dataset_id not set. Dataset selector will not be shown. Browser.js:535
> Annotation server error--maybe you forgot to login to the server? AnnotTrack.js:300
> 1.   TypeError
> "TypeError: Cannot read property 'xhr' of undefined at http://10.114.138.235/WebApollo/jbrowse/plugins/WebApollo/js/View/Track/AnnotTrack.js:301:59 at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:37:21) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- rejected at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- Error at Promise.then.promise.then (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:252:24) at declare.setViewInfo (http://10.114.138.235/WebApollo/j!
 browse/p
lugins/WebApollo/js/View/Track/AnnotTrack.js:255:16) at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2211:15) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 athttp://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2237:17 at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/Browser.js:1148:18) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 at runFactory (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1094:43) at execModule (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1223:5) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1258:6" instrumentation.js:18

> 1.   logErrorinstrumentation.js:18
> 'KeyboardEvent.keyLocation'' is deprecated. Please use 'KeyboardEvent.location' instead.
> event.returnValue is deprecated. Please use the standard event.preventDefault() instead.
>  
>
> Oct 29, 2014 8:39:31 AM org.apache.coyote.AbstractProtocol init
> INFO: Initializing ProtocolHandler ["http-bio-8080"]
> Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 339 ms
> Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardService startInternal
> INFO: Starting service Catalina
> Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardEngine startInternal
> INFO: Starting Servlet Engine: Apache Tomcat/7.0.26
> Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.HostConfig deployDescriptor
> INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/manager.xml
> Oct 29, 2014 8:39:44 AM org.apache.catalina.util.SessionIdGenerator createSecureRandom
> INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [12,606] milliseconds.
> Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDescriptor
> INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/host-manager.xml
> Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
> Oct 29, 2014 8:39:45 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory /var/lib/tomcat7/webapps/WebApollo
> Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.ContextConfig validateSecurityRoles
> INFO: WARNING: Security role name NobodyHasThisRole used in an <auth-constraint> without being defined in a <security-role>
> Oct 29, 2014 8:39:46 AM org.apache.coyote.AbstractProtocol start
> INFO: Starting ProtocolHandler ["http-bio-8080"]
> Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 13530 ms
>  
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Wednesday, October 29, 2014 12:12 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>  
>  
> And if you check your tomcat logs (catalina.out) for the same that would be helpful.
>  
> Nathan
>  
> On Oct 29, 2014, at 9:21 AM, Colin <[hidden email]> wrote:
>  
> Hi Brian,
> Would you be able to check your javascript console for any signs of errors?
>
> -Colin
>  
> On Wed, Oct 29, 2014 at 10:54 AM, Mack, Brian <[hidden email]> wrote:
> --------
> WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
> --------
>
> Nathan,
>
> I have all my maker predictions now copied to the user-created annotations, but for some reason WebApollo won't load the user-created annotations for the first 5 contigs. It just shows the spinning loading graphic. They are not much bigger than the other contigs that are loading fine. These contigs export to gff fine though. Any ideas?
>
> Brian
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Monday, October 27, 2014 3:24 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> > On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
> >
> > It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.
>
> Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.
>
> > So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?
>
> ;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.
>
> Colin said he’s going to take a look as well to double-check and may have an easier solution.
>
> In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.
>
> Nathan
>
> >
> > -----Original Message-----
> > From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > Sent: Monday, October 27, 2014 12:24 PM
> > To: [hidden email]
> > Subject: Re: [apollo] transfer entire track to user-created annotations
> >
> >
> > Based on that, I’m wondering if you’re running out of memory within perl.
> >
> > That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
> >
> > So, last two things to try:
> > 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> > 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.
> >
> > If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.
> >
> > I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
> >
> >
> > Nathan
> >
> >> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
> >>
> >> --------
> >> WARNING: At least one of the links in the message below goes to an IP
> >> address (e.g. 10.1.1.1), which could be malicious. To learn how to
> >> protect yourself, please go here: https://commons.lbl.gov/x/_591B
> >> --------
> >>
> >> Nathan,
> >>
> >> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
> >>
> >> Brian
> >>
> >> -----Original Message-----
> >> From: [hidden email]
> >> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >> Sent: Saturday, October 25, 2014 11:41 AM
> >> To: [hidden email]
> >> Subject: Re: [apollo] transfer entire track to user-created
> >> annotations
> >>
> >>
> >> I have a few more questions:
> >>
> >> 1 - does it always get stuck in the same place?
> >> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
> >>
> >> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.
> >>
> >> Nathan
> >>
> >>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
> >>>
> >>>
> >>> I hate to ask you, but where does it get stuck to on convert_features?
> >>>
> >>> Nathan
> >>>
> >>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
> >>>>
> >>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
> >>>>
> >>>> Brian
> >>>>
> >>>> 19 maker   gene    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
> >>>> 19 maker   mRNA    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
> >>>> 19 maker   exon    99910   100048  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   exon    100329  100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   CDS     99910   100048  .       +       0       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   CDS     100329  100435  .       +       2       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   gene    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
> >>>> 19 maker   mRNA    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
> >>>> 19 maker   exon    9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   exon    10124   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   five_prime_UTR  9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   five_prime_UTR  10124   10131   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   CDS     10132   10299   .       +       0       ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   three_prime_UTR 10300   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> ###
> >>>> 880        maker   gene    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
> >>>> 880        maker   mRNA    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
> >>>> 880        maker   exon    1276    1612    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> 880        maker   exon    1843    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> 880        maker   CDS     1276    1612    .       +       0       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> 880        maker   CDS     1843    2894    .       +       2       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> ###
> >>>> 24 maker   gene    3911    5179    .       +       .       ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
> >>>>
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: [hidden email]
> >>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >>>> Sent: Friday, October 24, 2014 1:26 PM
> >>>> To: [hidden email]
> >>>> Subject: Re: [apollo] transfer entire track to user-created
> >>>> annotations
> >>>>
> >>>>
> >>>> How large is your maker.gff file?   Can you see your annotation file growing?
> >>>>
> >>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?
> >>>>
> >>>> Can you add print statements to that script and see where it hangs.
> >>>>
> >>>> Nathan
> >>>>
> >>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
> >>>>>
> >>>>> Thanks. I am running it locally. I'm using the following command:
> >>>>>
> >>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
> >>>>> web_apollo_admin -U localhost:8080/WebApollo -i
> >>>>> /home/webapollo/maker.gff
> >>>>>
> >>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
> >>>>>
> >>>>> Brian
> >>>>>
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: [hidden email]
> >>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >>>>> Sent: Friday, October 24, 2014 12:06 PM
> >>>>> To: [hidden email]
> >>>>> Subject: Re: [apollo] transfer entire track to user-created
> >>>>> annotations
> >>>>>
> >>>>> Do you have the full command?
> >>>>>
> >>>>> If you are running it locally I would imagine it would be something like:
> >>>>>
> >>>>>
> >>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
> >>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
> >>>>> path/to/my.gff
> >>>>>
> >>>>> I’m wondering if you didn’t supply a user-name with -u?
> >>>>>
> >>>>> Nathan
> >>>>>
> >>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
> >>>>>>
> >>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
> >>>>>>
> >>>>>> [1] 9422
> >>>>>> [2] 9423
> >>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
> >>>>>>
> >>>>>> Am I using the correct URL format for this script?
> >>>>>>
> >>>>>> Brian
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: [hidden email]
> >>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
> >>>>>> To: [hidden email]
> >>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
> >>>>>> Subject: Re: [apollo] transfer entire track to user-created
> >>>>>> annotations
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
> >>>>>>> provided virtual machine. I’m unclear about the typical workflow
> >>>>>>> for saving your edited genes with the rest of the unedited annotations.
> >>>>>>> If I have a set of genes that I have modified
> >>>>>>>
> >>>>>>
> >>>>>> I think that that is what is most often done.
> >>>>>>
> >>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> In theory this should be in the last few releases based on this:
> >>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
> >>>>>>
> >>>>>> I think you use the script under tools:
> >>>>>>
> >>>>>> add_transcripts_from_gff3_to_annotations.pl
> >>>>>>
> >>>>>> Nathan
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Brian
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >
> >
> >
>


Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

Colin
It appears from the error message that this is WebApollo-2013-09-04, and is failing on this line:


Line 301 AnnotTrack.js: console.error("HTTP status code: ",ioArgs.xhr.status);

I think commenting out this line might be wise since this seems to have been removed in newer versions of WebApollo


Thanks,
-Colin

On Wed, Oct 29, 2014 at 3:57 PM, Nathan Dunn <[hidden email]> wrote:

Hmm . . it is failing roughly here (not sure the version, but I doubt its changed much).

If it works for some tracks and not others, I’m wondering if annotations for the other track simply did not exist.

> xhr(context_path + "/AnnotationEditorService", {
>     handleAs: "json",
>     data: '{ "track": "' + track.getUniqueTrackName() + '", "operation": "get_features" }',
>     method: "post"
> }).then(function(response, ioArgs) {
>     var responseFeatures = response.features;
> ….
> , function(response, ioArgs) { //
>     console.log("Annotation server error--maybe you forgot to login to the server?");
>     track.handleError({ responseText: response.response.text } );
> // . . .
>     return response; //
> });

If you repeat the process (for the first 5) does it do the same?

Nathan

> On Oct 29, 2014, at 11:35 AM, Mack, Brian <[hidden email]> wrote:
>
> --------
> WARNING: At least one of the links in the message below goes to an IP address (e.g.10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
> --------
>
> Nathan, I copied my javascript console and tomcat logs below. There seems to be an error in the javascript console.
>
> Brian
>
>
> loaded WebApollo plugin main.js:36
> Object
> Browser.js:1528
> In JBrowse configuration, datasets specified, but dataset_id not set. Dataset selector will not be shown. Browser.js:535
> Annotation server error--maybe you forgot to login to the server? AnnotTrack.js:300
> 1.   TypeError
> "TypeError: Cannot read property 'xhr' of undefined at http://10.114.138.235/WebApollo/jbrowse/plugins/WebApollo/js/View/Track/AnnotTrack.js:301:59 at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:37:21) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- rejected at signalDefer!
 red (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- Error at Promise.then.promise.then (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:252!
 :24) at declare.setViewInfo (http://10.114.138.235/WebApollo/j!
 browse/p
lugins/WebApollo/js/View/Track/AnnotTrack.js:255:16) at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2211:15) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 athttp://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2237:17 at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/Browser.js:1148:18) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 at runFactory (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1094:43) at execModule (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1223:5) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1258:6" instrumentation.js:18
> 1.   logErrorinstrumentation.js:18
> 'KeyboardEvent.keyLocation'' is deprecated. Please use 'KeyboardEvent.location' instead.
> event.returnValue is deprecated. Please use the standard event.preventDefault() instead.
>
>
> Oct 29, 2014 8:39:31 AM org.apache.coyote.AbstractProtocol init
> INFO: Initializing ProtocolHandler ["http-bio-8080"]
> Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 339 ms
> Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardService startInternal
> INFO: Starting service Catalina
> Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardEngine startInternal
> INFO: Starting Servlet Engine: Apache Tomcat/7.0.26
> Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.HostConfig deployDescriptor
> INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/manager.xml
> Oct 29, 2014 8:39:44 AM org.apache.catalina.util.SessionIdGenerator createSecureRandom
> INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [12,606] milliseconds.
> Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDescriptor
> INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/host-manager.xml
> Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
> Oct 29, 2014 8:39:45 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory /var/lib/tomcat7/webapps/WebApollo
> Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.ContextConfig validateSecurityRoles
> INFO: WARNING: Security role name NobodyHasThisRole used in an <auth-constraint> without being defined in a <security-role>
> Oct 29, 2014 8:39:46 AM org.apache.coyote.AbstractProtocol start
> INFO: Starting ProtocolHandler ["http-bio-8080"]
> Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 13530 ms
>
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Wednesday, October 29, 2014 12:12 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> And if you check your tomcat logs (catalina.out) for the same that would be helpful.
>
> Nathan
>
> On Oct 29, 2014, at 9:21 AM, Colin <[hidden email]> wrote:
>
> Hi Brian,
> Would you be able to check your javascript console for any signs of errors?
>
> -Colin
>
> On Wed, Oct 29, 2014 at 10:54 AM, Mack, Brian <[hidden email]> wrote:
> --------
> WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
> --------
>
> Nathan,
>
> I have all my maker predictions now copied to the user-created annotations, but for some reason WebApollo won't load the user-created annotations for the first 5 contigs. It just shows the spinning loading graphic. They are not much bigger than the other contigs that are loading fine. These contigs export to gff fine though. Any ideas?
>
> Brian
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> Sent: Monday, October 27, 2014 3:24 PM
> To: [hidden email]
> Subject: Re: [apollo] transfer entire track to user-created annotations
>
>
> > On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
> >
> > It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.
>
> Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.
>
> > So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?
>
> ;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.
>
> Colin said he’s going to take a look as well to double-check and may have an easier solution.
>
> In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.
>
> Nathan
>
> >
> > -----Original Message-----
> > From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > Sent: Monday, October 27, 2014 12:24 PM
> > To: [hidden email]
> > Subject: Re: [apollo] transfer entire track to user-created annotations
> >
> >
> > Based on that, I’m wondering if you’re running out of memory within perl.
> >
> > That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
> >
> > So, last two things to try:
> > 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> > 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.
> >
> > If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.
> >
> > I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
> >
> >
> > Nathan
> >
> >> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
> >>
> >> --------
> >> WARNING: At least one of the links in the message below goes to an IP
> >> address (e.g. 10.1.1.1), which could be malicious. To learn how to
> >> protect yourself, please go here: https://commons.lbl.gov/x/_591B
> >> --------
> >>
> >> Nathan,
> >>
> >> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
> >>
> >> Brian
> >>
> >> -----Original Message-----
> >> From: [hidden email]
> >> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >> Sent: Saturday, October 25, 2014 11:41 AM
> >> To: [hidden email]
> >> Subject: Re: [apollo] transfer entire track to user-created
> >> annotations
> >>
> >>
> >> I have a few more questions:
> >>
> >> 1 - does it always get stuck in the same place?
> >> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
> >>
> >> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.
> >>
> >> Nathan
> >>
> >>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
> >>>
> >>>
> >>> I hate to ask you, but where does it get stuck to on convert_features?
> >>>
> >>> Nathan
> >>>
> >>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
> >>>>
> >>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
> >>>>
> >>>> Brian
> >>>>
> >>>> 19 maker   gene    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
> >>>> 19 maker   mRNA    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
> >>>> 19 maker   exon    99910   100048  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   exon    100329  100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   CDS     99910   100048  .       +       0       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   CDS     100329  100435  .       +       2       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> >>>> 19 maker   gene    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
> >>>> 19 maker   mRNA    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
> >>>> 19 maker   exon    9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   exon    10124   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   five_prime_UTR  9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   five_prime_UTR  10124   10131   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   CDS     10132   10299   .       +       0       ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> 19 maker   three_prime_UTR 10300   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> >>>> ###
> >>>> 880        maker   gene    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
> >>>> 880        maker   mRNA    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
> >>>> 880        maker   exon    1276    1612    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> 880        maker   exon    1843    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> 880        maker   CDS     1276    1612    .       +       0       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> 880        maker   CDS     1843    2894    .       +       2       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> >>>> ###
> >>>> 24 maker   gene    3911    5179    .       +       .       ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
> >>>>
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: [hidden email]
> >>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >>>> Sent: Friday, October 24, 2014 1:26 PM
> >>>> To: [hidden email]
> >>>> Subject: Re: [apollo] transfer entire track to user-created
> >>>> annotations
> >>>>
> >>>>
> >>>> How large is your maker.gff file?   Can you see your annotation file growing?
> >>>>
> >>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?
> >>>>
> >>>> Can you add print statements to that script and see where it hangs.
> >>>>
> >>>> Nathan
> >>>>
> >>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
> >>>>>
> >>>>> Thanks. I am running it locally. I'm using the following command:
> >>>>>
> >>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
> >>>>> web_apollo_admin -U localhost:8080/WebApollo -i
> >>>>> /home/webapollo/maker.gff
> >>>>>
> >>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
> >>>>>
> >>>>> Brian
> >>>>>
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: [hidden email]
> >>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >>>>> Sent: Friday, October 24, 2014 12:06 PM
> >>>>> To: [hidden email]
> >>>>> Subject: Re: [apollo] transfer entire track to user-created
> >>>>> annotations
> >>>>>
> >>>>> Do you have the full command?
> >>>>>
> >>>>> If you are running it locally I would imagine it would be something like:
> >>>>>
> >>>>>
> >>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
> >>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
> >>>>> path/to/my.gff
> >>>>>
> >>>>> I’m wondering if you didn’t supply a user-name with -u?
> >>>>>
> >>>>> Nathan
> >>>>>
> >>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
> >>>>>>
> >>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
> >>>>>>
> >>>>>> [1] 9422
> >>>>>> [2] 9423
> >>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
> >>>>>>
> >>>>>> Am I using the correct URL format for this script?
> >>>>>>
> >>>>>> Brian
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: [hidden email]
> >>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> >>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
> >>>>>> To: [hidden email]
> >>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
> >>>>>> Subject: Re: [apollo] transfer entire track to user-created
> >>>>>> annotations
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
> >>>>>>> provided virtual machine. I’m unclear about the typical workflow
> >>>>>>> for saving your edited genes with the rest of the unedited annotations.
> >>>>>>> If I have a set of genes that I have modified
> >>>>>>>
> >>>>>>
> >>>>>> I think that that is what is most often done.
> >>>>>>
> >>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> In theory this should be in the last few releases based on this:
> >>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
> >>>>>>
> >>>>>> I think you use the script under tools:
> >>>>>>
> >>>>>> add_transcripts_from_gff3_to_annotations.pl
> >>>>>>
> >>>>>> Nathan
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Brian
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >
> >
> >
>



Reply | Threaded
Open this post in threaded view
|

Re: transfer entire track to user-created annotations

nathandunn

Also Brian (not sure why I didn’t notice this earlier) . . in the config.xml you can switch this to false, which may have solved your initial problem (not sure about the latter one):

<!-- set to false to use hybrid disk/memory store which provides a little slower performance
but uses a lot less memory - great for annotation rich genomes -->
<use_pure_memory_store>true</use_pure_memory_store>

Nathan


> On Oct 29, 2014, at 2:43 PM, Colin <[hidden email]> wrote:
>
> It appears from the error message that this is WebApollo-2013-09-04, and is failing on this line:
>
>
> Line 301 AnnotTrack.js: console.error("HTTP status code: ",ioArgs.xhr.status);
>
> I think commenting out this line might be wise since this seems to have been removed in newer versions of WebApollo
>
>
> Thanks,
> -Colin
>
> On Wed, Oct 29, 2014 at 3:57 PM, Nathan Dunn <[hidden email]> wrote:
>
> Hmm . . it is failing roughly here (not sure the version, but I doubt its changed much).
>
> If it works for some tracks and not others, I’m wondering if annotations for the other track simply did not exist.
>
> > xhr(context_path + "/AnnotationEditorService", {
> >     handleAs: "json",
> >     data: '{ "track": "' + track.getUniqueTrackName() + '", "operation": "get_features" }',
> >     method: "post"
> > }).then(function(response, ioArgs) {
> >     var responseFeatures = response.features;
> > ….
> > , function(response, ioArgs) { //
> >     console.log("Annotation server error--maybe you forgot to login to the server?");
> >     track.handleError({ responseText: response.response.text } );
> > // . . .
> >     return response; //
> > });
>
> If you repeat the process (for the first 5) does it do the same?
>
> Nathan
>
> > On Oct 29, 2014, at 11:35 AM, Mack, Brian <[hidden email]> wrote:
> >
> > --------
> > WARNING: At least one of the links in the message below goes to an IP address (e.g.10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
> > --------
> >
> > Nathan, I copied my javascript console and tomcat logs below. There seems to be an error in the javascript console.
> >
> > Brian
> >
> >
> > loaded WebApollo plugin main.js:36
> > Object
> > Browser.js:1528
> > In JBrowse configuration, datasets specified, but dataset_id not set. Dataset selector will not be shown. Browser.js:535
> > Annotation server error--maybe you forgot to login to the server? AnnotTrack.js:300
> > 1.   TypeError
> > "TypeError: Cannot read property 'xhr' of undefined at http://10.114.138.235/WebApollo/jbrowse/plugins/WebApollo/js/View/Track/AnnotTrack.js:301:59 at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:37:21) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- rejected at signalDefer!
>  red (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:58:4) at signalWaiting (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:28:4) at reject.reject (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:219:5) at signalDeferred (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:84:15) at signalListener (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:55:5) ---------------------------------------- Error at Promise.then.promise.then (http://10.114.138.235/WebApollo/jbrowse/src/dojo/Deferred.js:252!
>  :24) at declare.setViewInfo (http://10.114.138.235/WebApollo/j!
>  browse/p
> lugins/WebApollo/js/View/Track/AnnotTrack.js:255:16) at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2211:15) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 athttp://10.114.138.235/WebApollo/jbrowse/src/JBrowse/GenomeView.js:2237:17 at null.<anonymous> (http://10.114.138.235/WebApollo/jbrowse/src/JBrowse/Browser.js:1148:18) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/_base/lang.js:375:55 at runFactory (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1094:43) at execModule (http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1223:5) at http://10.114.138.235/WebApollo/jbrowse/src/dojo/dojo.js:1258:6" instrumentation.js:18
> > 1.   logErrorinstrumentation.js:18
> > 'KeyboardEvent.keyLocation'' is deprecated. Please use 'KeyboardEvent.location' instead.
> > event.returnValue is deprecated. Please use the standard event.preventDefault() instead.
> >
> >
> > Oct 29, 2014 8:39:31 AM org.apache.coyote.AbstractProtocol init
> > INFO: Initializing ProtocolHandler ["http-bio-8080"]
> > Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.Catalina load
> > INFO: Initialization processed in 339 ms
> > Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardService startInternal
> > INFO: Starting service Catalina
> > Oct 29, 2014 8:39:31 AM org.apache.catalina.core.StandardEngine startInternal
> > INFO: Starting Servlet Engine: Apache Tomcat/7.0.26
> > Oct 29, 2014 8:39:31 AM org.apache.catalina.startup.HostConfig deployDescriptor
> > INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/manager.xml
> > Oct 29, 2014 8:39:44 AM org.apache.catalina.util.SessionIdGenerator createSecureRandom
> > INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [12,606] milliseconds.
> > Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDescriptor
> > INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/host-manager.xml
> > Oct 29, 2014 8:39:44 AM org.apache.catalina.startup.HostConfig deployDirectory
> > INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
> > Oct 29, 2014 8:39:45 AM org.apache.catalina.startup.HostConfig deployDirectory
> > INFO: Deploying web application directory /var/lib/tomcat7/webapps/WebApollo
> > Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.ContextConfig validateSecurityRoles
> > INFO: WARNING: Security role name NobodyHasThisRole used in an <auth-constraint> without being defined in a <security-role>
> > Oct 29, 2014 8:39:46 AM org.apache.coyote.AbstractProtocol start
> > INFO: Starting ProtocolHandler ["http-bio-8080"]
> > Oct 29, 2014 8:39:46 AM org.apache.catalina.startup.Catalina start
> > INFO: Server startup in 13530 ms
> >
> > From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > Sent: Wednesday, October 29, 2014 12:12 PM
> > To: [hidden email]
> > Subject: Re: [apollo] transfer entire track to user-created annotations
> >
> >
> > And if you check your tomcat logs (catalina.out) for the same that would be helpful.
> >
> > Nathan
> >
> > On Oct 29, 2014, at 9:21 AM, Colin <[hidden email]> wrote:
> >
> > Hi Brian,
> > Would you be able to check your javascript console for any signs of errors?
> >
> > -Colin
> >
> > On Wed, Oct 29, 2014 at 10:54 AM, Mack, Brian <[hidden email]> wrote:
> > --------
> > WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
> > --------
> >
> > Nathan,
> >
> > I have all my maker predictions now copied to the user-created annotations, but for some reason WebApollo won't load the user-created annotations for the first 5 contigs. It just shows the spinning loading graphic. They are not much bigger than the other contigs that are loading fine. These contigs export to gff fine though. Any ideas?
> >
> > Brian
> >
> > -----Original Message-----
> > From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > Sent: Monday, October 27, 2014 3:24 PM
> > To: [hidden email]
> > Subject: Re: [apollo] transfer entire track to user-created annotations
> >
> >
> > > On Oct 27, 2014, at 12:07 PM, Mack, Brian <[hidden email]> wrote:
> > >
> > > It seems like Tomcat is running out of memory. Whenever I run the script, the memory used by tomcat7 goes to 1200Mb and then hangs. I have 4Gb total memory. I took the first Mb of my gff file and was able to successfully run it. However running the script on the second Mb of my gff file hung when the memory went to 1200Mb. It didn't release the memory from the first run of the script. I was able to get the second portion of my gff file to work when I stopped and started the tomcat7 service.
> >
> > Holy wow.  I’m really glad you checked for tomcat as changed to the script wouldn’t have done anything.  Some of our larger users run with about 2-4 GB’s.
> >
> > > So I guess splitting the gff file into smaller parts and running the script after stopping and starting Tomcat will solve my problem. But, if I split my gff file how you suggested, won't it cause errors if a gene or contig is split into different files?
> >
> > ;)  Yes, sorry.   Yo should be able to break it up by seqids / chromosome (looks like you have around 800).  You can modify the split_gff_by_source to use column 0 instead of column 1 and then just run them individually.
> >
> > Colin said he’s going to take a look as well to double-check and may have an easier solution.
> >
> > In the near/long-term (early next year) we are refactoring the back-end to avoid this type of memory problem.  I wish I could get it out sooner.
> >
> > Nathan
> >
> > >
> > > -----Original Message-----
> > > From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > > Sent: Monday, October 27, 2014 12:24 PM
> > > To: [hidden email]
> > > Subject: Re: [apollo] transfer entire track to user-created annotations
> > >
> > >
> > > Based on that, I’m wondering if you’re running out of memory within perl.
> > >
> > > That section of code is building a stack of features to write out in a fairly recursive fashion and only writes out at the very end meaning that memory is likely piling up.   The other possibility, though unlikely scenario is that you have a loop in your annotations (not sure how this would happen), but I would expect things to die at the same spot each time.
> > >
> > > So, last two things to try:
> > > 1 - when it gets stuck (again, sorry), can you take a look at memory and see where you’re at using top.
> > > 2 - can you arbitrarily split up your script into multiple GFF 3 files and run them individually?  (split -l1183 mylarge.gff ) and then run them one by one in a script.
> > >
> > > If #1 indicates a memory problem and #2 works, I will open a bug to fix this by writing out more.
> > >
> > > I expect our GFF files will be getting larger in the future, so its really important for this to work properly.  Sorry you had to be the one to run into it.
> > >
> > >
> > > Nathan
> > >
> > >> On Oct 27, 2014, at 7:52 AM, Mack, Brian <[hidden email]> wrote:
> > >>
> > >> --------
> > >> WARNING: At least one of the links in the message below goes to an IP
> > >> address (e.g. 10.1.1.1), which could be malicious. To learn how to
> > >> protect yourself, please go here: https://commons.lbl.gov/x/_591B
> > >> --------
> > >>
> > >> Nathan,
> > >>
> > >> I sent you my gff3 file and fasta. It does not get stuck at the same place in the gff3  file each time. I've added more print statements and it seems to be stopping right after the call to "process_gff_entry" in "process_gff" but before the "write_features" call immediately after (between lines 211 and 212).
> > >>
> > >> Brian
> > >>
> > >> -----Original Message-----
> > >> From: [hidden email]
> > >> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > >> Sent: Saturday, October 25, 2014 11:41 AM
> > >> To: [hidden email]
> > >> Subject: Re: [apollo] transfer entire track to user-created
> > >> annotations
> > >>
> > >>
> > >> I have a few more questions:
> > >>
> > >> 1 - does it always get stuck in the same place?
> > >> 2 - if you just run that section of GFF3, does it get stuck on a particular line in convert_features (below)?
> > >>
> > >> Its possible a parsing error or a transaction error.  If you give me access to your gff3 file I can also run it, as well.
> > >>
> > >> Nathan
> > >>
> > >>> On Oct 24, 2014, at 6:12 PM, Nathan Dunn <[hidden email]> wrote:
> > >>>
> > >>>
> > >>> I hate to ask you, but where does it get stuck to on convert_features?
> > >>>
> > >>> Nathan
> > >>>
> > >>>> On Oct 24, 2014, at 1:55 PM, Mack, Brian <[hidden email]> wrote:
> > >>>>
> > >>>> Thanks for the suggestion about using the print statements. The script reaches the processing step and does not get to the writing. The last transcript that it sent to the "convert_feature" function is "maker-19-augustus-gene-0.45-mRNA-1". It hangs at that point and doesn't process any more genes. This is about 20% into my 15Mb gff file. I copied this gene and the one surrounding it below. I don't see anything unique about it.
> > >>>>
> > >>>> Brian
> > >>>>
> > >>>> 19 maker   gene    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62
> > >>>> 19 maker   mRNA    99910   100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1;Parent=maker-19-augustus-gene-0.62;Name=maker-19-augustus-gene-0.62-mRNA-1;_AED=0.53;_eAED=0.53;_QI=0|0|0|1|0|0|2|0|81
> > >>>> 19 maker   exon    99910   100048  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38175;Parent=maker-19-augustus-gene-0.62-mRNA-1
> > >>>> 19 maker   exon    100329  100435  .       +       .       ID=maker-19-augustus-gene-0.62-mRNA-1:exon:38176;Parent=maker-19-augustus-gene-0.62-mRNA-1
> > >>>> 19 maker   CDS     99910   100048  .       +       0       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> > >>>> 19 maker   CDS     100329  100435  .       +       2       ID=maker-19-augustus-gene-0.62-mRNA-1:cds;Parent=maker-19-augustus-gene-0.62-mRNA-1
> > >>>> 19 maker   gene    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45
> > >>>> 19 maker   mRNA    9939    10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1;Parent=maker-19-augustus-gene-0.45;Name=maker-19-augustus-gene-0.45-mRNA-1;_AED=0.51;_eAED=1.00;_QI=43|0|0|1|0|0|2|167|55
> > >>>> 19 maker   exon    9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38177;Parent=maker-19-augustus-gene-0.45-mRNA-1
> > >>>> 19 maker   exon    10124   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:exon:38178;Parent=maker-19-augustus-gene-0.45-mRNA-1
> > >>>> 19 maker   five_prime_UTR  9939    9973    .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> > >>>> 19 maker   five_prime_UTR  10124   10131   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:five_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> > >>>> 19 maker   CDS     10132   10299   .       +       0       ID=maker-19-augustus-gene-0.45-mRNA-1:cds;Parent=maker-19-augustus-gene-0.45-mRNA-1
> > >>>> 19 maker   three_prime_UTR 10300   10466   .       +       .       ID=maker-19-augustus-gene-0.45-mRNA-1:three_prime_utr;Parent=maker-19-augustus-gene-0.45-mRNA-1
> > >>>> ###
> > >>>> 880        maker   gene    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0
> > >>>> 880        maker   mRNA    1276    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1;Parent=genemark-880-processed-gene-0.0;Name=genemark-880-processed-gene-0.0-mRNA-1;_AED=0.16;_eAED=0.16;_QI=0|0|0|1|1|1|2|0|462
> > >>>> 880        maker   exon    1276    1612    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26708;Parent=genemark-880-processed-gene-0.0-mRNA-1
> > >>>> 880        maker   exon    1843    2894    .       +       .       ID=genemark-880-processed-gene-0.0-mRNA-1:exon:26709;Parent=genemark-880-processed-gene-0.0-mRNA-1
> > >>>> 880        maker   CDS     1276    1612    .       +       0       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> > >>>> 880        maker   CDS     1843    2894    .       +       2       ID=genemark-880-processed-gene-0.0-mRNA-1:cds;Parent=genemark-880-processed-gene-0.0-mRNA-1
> > >>>> ###
> > >>>> 24 maker   gene    3911    5179    .       +       .       ID=maker-24-pred_gff_JCVI-gene-0.141;Name=maker-24-pred_gff_JCVI-gene-0.141;Alias='eukaryotic translation initiation factor 5, putative'
> > >>>>
> > >>>>
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: [hidden email]
> > >>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > >>>> Sent: Friday, October 24, 2014 1:26 PM
> > >>>> To: [hidden email]
> > >>>> Subject: Re: [apollo] transfer entire track to user-created
> > >>>> annotations
> > >>>>
> > >>>>
> > >>>> How large is your maker.gff file?   Can you see your annotation file growing?
> > >>>>
> > >>>> I’m wondering if its I/O bound writing to disk or just hanging on connecting to postgres or your server?
> > >>>>
> > >>>> Can you add print statements to that script and see where it hangs.
> > >>>>
> > >>>> Nathan
> > >>>>
> > >>>>> On Oct 24, 2014, at 10:55 AM, Mack, Brian <[hidden email]> wrote:
> > >>>>>
> > >>>>> Thanks. I am running it locally. I'm using the following command:
> > >>>>>
> > >>>>> add_transcripts_from_gff3_to_annotations.pl -u web_apollo_admin -p
> > >>>>> web_apollo_admin -U localhost:8080/WebApollo -i
> > >>>>> /home/webapollo/maker.gff
> > >>>>>
> > >>>>> But, it's taking a very long time to complete and when I look at 'top', there doesn't seem to be anything going on.
> > >>>>>
> > >>>>> Brian
> > >>>>>
> > >>>>>
> > >>>>> -----Original Message-----
> > >>>>> From: [hidden email]
> > >>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > >>>>> Sent: Friday, October 24, 2014 12:06 PM
> > >>>>> To: [hidden email]
> > >>>>> Subject: Re: [apollo] transfer entire track to user-created
> > >>>>> annotations
> > >>>>>
> > >>>>> Do you have the full command?
> > >>>>>
> > >>>>> If you are running it locally I would imagine it would be something like:
> > >>>>>
> > >>>>>
> > >>>>> ./tools/data/add_transcripts_from_gff3_to_annotations.pl -U
> > >>>>> localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin -i
> > >>>>> path/to/my.gff
> > >>>>>
> > >>>>> I’m wondering if you didn’t supply a user-name with -u?
> > >>>>>
> > >>>>> Nathan
> > >>>>>
> > >>>>>> On Oct 24, 2014, at 5:46 AM, Mack, Brian <[hidden email]> wrote:
> > >>>>>>
> > >>>>>> I'm having trouble with the add_transcripts_from_gff3_to_annotations.pl script. I'm not sure what to put for the "-U" argument. I have tried "-U http://127.0.0.1/WebApollo" and it just hangs and does not add any transcripts. I have also tried the full url, "-U jbrowse/?loc=1%3A8554..21872&tracks=DNA%2CAnnotations&highlight=", but it returns the following message and also hangs:
> > >>>>>>
> > >>>>>> [1] 9422
> > >>>>>> [2] 9423
> > >>>>>> webapollo@webapollo-ubuntu:~/webapollo$ malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE html>\n<ht...") at /home/webapollo/webapollo/WebApollo-2013-09-04/tools/data/add_transcripts_from_gff3_to_annotations.pl line 169.
> > >>>>>>
> > >>>>>> Am I using the correct URL format for this script?
> > >>>>>>
> > >>>>>> Brian
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> -----Original Message-----
> > >>>>>> From: [hidden email]
> > >>>>>> [mailto:[hidden email]] On Behalf Of Nathan Dunn
> > >>>>>> Sent: Tuesday, October 21, 2014 3:36 PM
> > >>>>>> To: [hidden email]
> > >>>>>> Cc: Monica C. Munoz-Torres; Monica Munoz-Torres; Colin Diesh
> > >>>>>> Subject: Re: [apollo] transfer entire track to user-created
> > >>>>>> annotations
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Oct 21, 2014, at 11:12 AM, Mack, Brian <[hidden email]> wrote:
> > >>>>>>>
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I’ve successfully loaded my Maker gff3 into WebApollo on the
> > >>>>>>> provided virtual machine. I’m unclear about the typical workflow
> > >>>>>>> for saving your edited genes with the rest of the unedited annotations.
> > >>>>>>> If I have a set of genes that I have modified
> > >>>>>>>
> > >>>>>>
> > >>>>>> I think that that is what is most often done.
> > >>>>>>
> > >>>>>>> Or, is there a way to promote the entire Maker gene prediction track to the user-created annotations? I saw this mentioned in a previous threadhttp://gmod.827538.n3.nabble.com/user-created-annotation-area-td4030085.html#a4030087), but I couldn’t find it in the documentation.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>> In theory this should be in the last few releases based on this:
> > >>>>>> http://gmod.org/wiki/News/New_WebApollo_Release_(2013/11/22))
> > >>>>>>
> > >>>>>> I think you use the script under tools:
> > >>>>>>
> > >>>>>> add_transcripts_from_gff3_to_annotations.pl
> > >>>>>>
> > >>>>>> Nathan
> > >>>>>>
> > >>>>>>> Thanks,
> > >>>>>>>
> > >>>>>>> Brian
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >
> > >
> > >
> >
>
>
>


12