Re: Problems with collections

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

Julie Sullivan-2
Hi Paulo,

For a reverse reference to work, the labels have to match.

If you have "genes" as the collection, then "genes" (not "gene"!) has to
be the value of the reverse reference to that collection.

Does that make sense?

Here are some docs:

https://intermine.readthedocs.io/en/latest/data-model/model/

I would correct your data model, run a clean and rebuild and try the
query again.

Cheers,
Julie

PS. I'm CCing the dev list because I think this might help other people
too!

On 12/11/2018 16:43, Paulo Nuin wrote:

> Hi
>
> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>
> Thanks
>
> Paulo
>
>
>
>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>
>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>
>> Is that a typo?
>>
>>
>>
>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>> Hi Julie
>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>> Our project addition file is on
>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>> and the Allele class has
>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>> while the Gene has
>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>> but gk962622 just shows one gene while it should be linked to many.
>>> Any help appreciated
>>> Thanks
>>> Paulo
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

Julie Sullivan-2
Paulo,

(CCing the dev list for help.)

Yes your data model looks correct now!

What does the data look like after your build? Are both collections
(affectedByAlleles, affectsGenes) empty?

I would check your data parser. Are you setting the collections
correctly? For example, you should have one of these statements:

        allele.addToCollection("affectsGenes", gene)
        gene.addToCollection("affectedByAlleles", allele)

Here are some docs on many-to-many relationships:

https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship

Does that help?
Julie

On 20/11/2018 13:48, Paulo Nuin wrote:

> Hi Julie
>
> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>
>
>          <class name="Gene" is-interface="true">
>                  <attribute name="operon" type="java.lang.String"/>
>                  <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>                  <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>                  <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>                  <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>                  <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>                  <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>                  <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>                  <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>          </class>
>
>
>   <class name="Allele" is-interface="true">
>            <attribute name="naturalVariant" type="java.lang.String" />
>            <attribute name="method" type="java.lang.String" />
>            <attribute name="type" type="java.lang.String" />
>            <attribute name="publicName" type="java.lang.String" />
>            <attribute name="otherName" type="java.lang.String" />
>            <attribute name="KOAllele" type="java.lang.String" />
>            <attribute name="NBPAllele" type="java.lang.String" />
>            <attribute name="nemaAllele" type="java.lang.String" />
>            <attribute name="amber_UAG" type="java.lang.String" />
>            <attribute name="ochre_UAA" type="java.lang.String" />
>            <attribute name="opal_UGA" type="java.lang.String" />
>            <attribute name="donorSpliceSite" type="java.lang.String" />
>            <attribute name="acceptorSpliceSite" type="java.lang.String" />
>            <attribute name="productionMethod" type="java.lang.String" />
> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>            <attribute name="status" type="java.lang.String" />
> <!--      <attribute name="affects" type="java.lang.String" /> -->
>            <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>            <attribute name="positiveClone" type="java.lang.String" />
>            <attribute name="typeOfMutation" type="java.lang.String" />
>            <attribute name="sequenceStatus" type="java.lang.String" />
>            <attribute name="KOConsortiumAllele" type="java.lang.String" />
>            <attribute name="missenseChange" type="java.lang.String" />
>            <attribute name="mutagen" type="java.lang.String" />
>            <attribute name="natureOfVariation" type="java.lang.String" />
>            <attribute name="interpolatedMapPosition" type="java.lang.String" />
>            <attribute name="species" type="java.lang.String" />
>            <attribute name="rearrangement" type="java.lang.String" />
>            <attribute name="linkedTo" type="java.lang.String" />
>            <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>            <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>            <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>            <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>            <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>            <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>            <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>          </class>
>
> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>
>
>
> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
> [integrate] JDJDDEBUG:: =======================
> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
> [integrate] JDJDDEBUG:: fieldname=opal_UGA
> [integrate] JDJDDEBUG:: xpathvalue:
> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
> [integrate] JDJDDEBUG:: =======================
> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
> [integrate] JDJDDEBUG:: fieldname=gene
> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
> [integrate] JDJDDEBUG:: N:1
>
> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>
> Thanks again
>
> Paulo
>
>
>
>
>
>
>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>
>> Hi Paulo,
>>
>> For a reverse reference to work, the labels have to match.
>>
>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>
>> Does that make sense?
>>
>> Here are some docs:
>>
>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>
>> I would correct your data model, run a clean and rebuild and try the query again.
>>
>> Cheers,
>> Julie
>>
>> PS. I'm CCing the dev list because I think this might help other people too!
>>
>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>> Hi
>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>> Thanks
>>> Paulo
>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>
>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>
>>>> Is that a typo?
>>>>
>>>>
>>>>
>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>> Hi Julie
>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>> Our project addition file is on
>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>> and the Allele class has
>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>> while the Gene has
>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>> Any help appreciated
>>>>> Thanks
>>>>> Paulo
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

Paulo Nuin
Hi Julie

Here what an Allele item looks like (all *s are NULL)

-[ RECORD 1 ]---------------+-------------------------------
othername                   | cewivar00722050
linkedto                    | *
interpolatedmapposition     | *
lastupdated                 | *
sequencestatus              | Sequenced
koallele                    | *
nemaallele                  | *
natureofvariation           | *
rearrangement               | *
status                      | Live
intermine_method            | *
nbpallele                   | false
mutagen                     | *
amber_uag                   | *
publicname                  | *
type                        | Natural_variant
species                     | Caenorhabditis elegans
id                          | 1000004
name                        | *
productionmethod            | *
missensechange              | *
nemagenetagconsortiumallele | false
secondaryidentifier         | *
positiveclone               | *
naturalvariant              | *
opal_uga                    | *
ochre_uaa                   | *
typeofmutation              | Deletion
donorsplicesite             | *
symbol                      | WBVar01994782
acceptorsplicesite          | *
primaryidentifier           | WBVar01994782
detectionmethod             | *
koconsortiumallele          | false
organismid                  | *
sequenceontologytermid      | 1000003
geneid                      | 1000005
class                       | org.intermine.model.bio.Allele


And I have

intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
 affectedbyalleles | affectsgenes
-------------------+--------------
(0 rows) 

Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at 



I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection. 


[integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
[integrate] JDJDDEBUG:: cast type: null
[integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
[integrate] JDJDDEBUG:: gene.primaryIdentifier
[integrate] JDJDDEBUG:: passed
[integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
[integrate] JDJDDEBUG:: new1 primaryIdentifier
[integrate] JDJDDEBUG:: new2 primaryIdentifier
[integrate] JDJDDEBUG:: false

I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.

Cheers
Paulo




On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email]> wrote:

Paulo,

(CCing the dev list for help.)

Yes your data model looks correct now!

What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?

I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:

allele.addToCollection("affectsGenes", gene)
gene.addToCollection("affectedByAlleles", allele)

Here are some docs on many-to-many relationships:

https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship

Does that help?
Julie

On 20/11/2018 13:48, Paulo Nuin wrote:
Hi Julie
Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
        <class name="Gene" is-interface="true">
                <attribute name="operon" type="java.lang.String"/>
                <!--<reference name="referenceAllele" referenced-type="Allele" />-->
                <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
                <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
                <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
                <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
—>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
                <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
                <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
                <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
        </class>
 <class name="Allele" is-interface="true">
          <attribute name="naturalVariant" type="java.lang.String" />
          <attribute name="method" type="java.lang.String" />
          <attribute name="type" type="java.lang.String" />
          <attribute name="publicName" type="java.lang.String" />
          <attribute name="otherName" type="java.lang.String" />
          <attribute name="KOAllele" type="java.lang.String" />
          <attribute name="NBPAllele" type="java.lang.String" />
          <attribute name="nemaAllele" type="java.lang.String" />
          <attribute name="amber_UAG" type="java.lang.String" />
          <attribute name="ochre_UAA" type="java.lang.String" />
          <attribute name="opal_UGA" type="java.lang.String" />
          <attribute name="donorSpliceSite" type="java.lang.String" />
          <attribute name="acceptorSpliceSite" type="java.lang.String" />
          <attribute name="productionMethod" type="java.lang.String" />
<!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
          <attribute name="status" type="java.lang.String" />
<!--      <attribute name="affects" type="java.lang.String" /> -->
          <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
          <attribute name="positiveClone" type="java.lang.String" />
          <attribute name="typeOfMutation" type="java.lang.String" />
          <attribute name="sequenceStatus" type="java.lang.String" />
          <attribute name="KOConsortiumAllele" type="java.lang.String" />
          <attribute name="missenseChange" type="java.lang.String" />
          <attribute name="mutagen" type="java.lang.String" />
          <attribute name="natureOfVariation" type="java.lang.String" />
          <attribute name="interpolatedMapPosition" type="java.lang.String" />
          <attribute name="species" type="java.lang.String" />
          <attribute name="rearrangement" type="java.lang.String" />
          <attribute name="linkedTo" type="java.lang.String" />
          <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
          <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
—>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
          <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
          <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
          <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
          <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
          <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
        </class>
As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
[integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
[integrate] JDJDDEBUG:: suffix:primaryIdentifier
[integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
[integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
[integrate] JDJDDEBUG:: =======================
[integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
[integrate] JDJDDEBUG:: fieldname=opal_UGA
[integrate] JDJDDEBUG:: xpathvalue:
[integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
[integrate] JDJDDEBUG:: =======================
[integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
[integrate] JDJDDEBUG:: suffix:primaryIdentifier
[integrate] JDJDDEBUG:: fieldname=gene
[integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
[integrate] JDJDDEBUG:: N:1
Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
Thanks again
Paulo
On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:

Hi Paulo,

For a reverse reference to work, the labels have to match.

If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.

Does that make sense?

Here are some docs:

https://intermine.readthedocs.io/en/latest/data-model/model/

I would correct your data model, run a clean and rebuild and try the query again.

Cheers,
Julie

PS. I'm CCing the dev list because I think this might help other people too!

On 12/11/2018 16:43, Paulo Nuin wrote:
Hi
I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
Thanks
Paulo
On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:

Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".

Is that a typo?



On 12/11/2018 15:37, Paulo Nuin wrote:
Hi Julie
We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
Our project addition file is on
https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
and the Allele class has
<collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
while the Gene has
<collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
<query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
but gk962622 just shows one gene while it should be linked to many.
Any help appreciated
Thanks
Paulo


_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

sergio contrino-2
hi paulo,
interesting your generic approach: do your system works for CDSs and
trascripts to allele (i.e. it is a problem only for the gene allele
relationship)?
could you send a link also to the mapping file(s)?
thanks
sergio


On 21/11/2018 14:18, Paulo Nuin wrote:

> Hi Julie
>
> Here what an Allele item looks like (all *s are NULL)
>
> -[ RECORD 1 ]---------------+-------------------------------
> othername                   | cewivar00722050
> linkedto                    | *
> interpolatedmapposition     | *
> lastupdated                 | *
> sequencestatus              | Sequenced
> koallele                    | *
> nemaallele                  | *
> natureofvariation           | *
> rearrangement               | *
> status                      | Live
> intermine_method            | *
> nbpallele                   | false
> mutagen                     | *
> amber_uag                   | *
> publicname                  | *
> type                        | Natural_variant
> species                     | Caenorhabditis elegans
> id                          | 1000004
> name                        | *
> productionmethod            | *
> missensechange              | *
> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->


> secondaryidentifier         | *
> positiveclone               | *
> naturalvariant              | *
> opal_uga                    | *
> ochre_uaa                   | *
> typeofmutation              | Deletion
> donorsplicesite             | *
> symbol                      | WBVar01994782
> acceptorsplicesite          | *
> primaryidentifier           | WBVar01994782
> detectionmethod             | *
> koconsortiumallele          | false
> organismid                  | *
> sequenceontologytermid      | 1000003
> geneid                      | 1000005
> class                       | org.intermine.model.bio.Allele
>
>
> And I have
>
> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>   affectedbyalleles | affectsgenes
> -------------------+--------------
> (0 rows)
>
> Our parser is generic for 10 or more Xml files and most of other
> collections are being set properly. We have 38 testing queries, most of
> them testing collections and this is the only that fails. Parser is at
>
> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>
>
> I see this from the loader log. Gene is being cast as NULL. I think I
> need to create a collection Gene instead of having the affectedByAllele
> collection.
>
>
> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
> [integrate] JDJDDEBUG:: cast type: null
> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
> [integrate] JDJDDEBUG:: gene.primaryIdentifier
> [integrate] JDJDDEBUG:: passed
> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
> [integrate] JDJDDEBUG:: new1 primaryIdentifier
> [integrate] JDJDDEBUG:: new2 primaryIdentifier
> [integrate] JDJDDEBUG:: false
>
> I have some other debugging options, will try a new collection and
> uncomment the debug statements and get back to you.
>
> Cheers
> Paulo
>
>
>
>
>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> Paulo,
>>
>> (CCing the dev list for help.)
>>
>> Yes your data model looks correct now!
>>
>> What does the data look like after your build? Are both collections
>> (affectedByAlleles, affectsGenes) empty?
>>
>> I would check your data parser. Are you setting the collections
>> correctly? For example, you should have one of these statements:
>>
>> allele.addToCollection("affectsGenes", gene)
>> gene.addToCollection("affectedByAlleles", allele)
>>
>> Here are some docs on many-to-many relationships:
>>
>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>
>> Does that help?
>> Julie
>>
>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>> Hi Julie
>>> Still have\ing errors with this. I made some changes to my Gene and
>>> Allele classes and affectedByAlleles is linked as a collection to
>>> affectsGenes. There are no plural or singular differences
>>>         <class name="Gene" is-interface="true">
>>>                 <attribute name="operon" type="java.lang.String"/>
>>>                 <!--<reference name="referenceAllele"
>>> referenced-type="Allele" />-->
>>>                 <collection name="strains" referenced-type="Strain"
>>> reverse-reference="genes"/>
>>>                 <collection name="expressionPatterns"
>>> referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>                 <collection name="expressionClusters"
>>> referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>                 <collection name="regulatesExprCluster"
>>> referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele"
>>> reverse-reference="affectsGenes" />
>>>                 <collection name="RNAiResult" referenced-type="RNAi"
>>> reverse-reference="inhibitsGene"/>
>>>                 <collection name="geneClass"
>>>  referenced-type="GeneClass" reverse-reference="genes"/>
>>>                 <collection name="formerGeneClass"
>>>  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>         </class>
>>>  <class name="Allele" is-interface="true">
>>>           <attribute name="naturalVariant" type="java.lang.String" />
>>>           <attribute name="method" type="java.lang.String" />
>>>           <attribute name="type" type="java.lang.String" />
>>>           <attribute name="publicName" type="java.lang.String" />
>>>           <attribute name="otherName" type="java.lang.String" />
>>>           <attribute name="KOAllele" type="java.lang.String" />
>>>           <attribute name="NBPAllele" type="java.lang.String" />
>>>           <attribute name="nemaAllele" type="java.lang.String" />
>>>           <attribute name="amber_UAG" type="java.lang.String" />
>>>           <attribute name="ochre_UAA" type="java.lang.String" />
>>>           <attribute name="opal_UGA" type="java.lang.String" />
>>>           <attribute name="donorSpliceSite" type="java.lang.String" />
>>>           <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>           <attribute name="productionMethod" type="java.lang.String" />
>>> <!--      <attribute name="sequenceDetails" type="java.lang.String"
>>> /> -->
>>>           <attribute name="status" type="java.lang.String" />
>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>           <attribute name="NemaGENETAGConsortiumAllele"
>>> type="java.lang.String" />          <attribute name="detectionMethod"
>>> type="java.lang.String" />
>>>           <attribute name="positiveClone" type="java.lang.String" />
>>>           <attribute name="typeOfMutation" type="java.lang.String" />
>>>           <attribute name="sequenceStatus" type="java.lang.String" />
>>>           <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>           <attribute name="missenseChange" type="java.lang.String" />
>>>           <attribute name="mutagen" type="java.lang.String" />
>>>           <attribute name="natureOfVariation" type="java.lang.String" />
>>>           <attribute name="interpolatedMapPosition"
>>> type="java.lang.String" />
>>>           <attribute name="species" type="java.lang.String" />
>>>           <attribute name="rearrangement" type="java.lang.String" />
>>>           <attribute name="linkedTo" type="java.lang.String" />
>>>           <collection name="phenotype" referenced-type="Phenotype"
>>> reverse-reference="phenotypeAllele" />
>>>           <collection name="phenotypeNotObserved"
>>> referenced-type="Phenotype"
>>> reverse-reference="phenotypeAlleleNotObserved" />
>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene"
>>> reverse-reference="affectedByAlleles" />
>>>           <!--<collection name="affectsCDSs" referenced-type="CDS"
>>> reverse-reference="affectedByAlleles" />
>>>           <collection name="affectsTranscripts"
>>> referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>           <collection name="strains" referenced-type="Strain"
>>> reverse-reference="variations" /> -->
>>>           <collection name="fromLabs" referenced-type="Laboratory"
>>> reverse-reference="variations" />
>>>           <collection name="geneClass" referenced-type="GeneClass"
>>> reverse-reference="variations" />
>>>         </class>
>>> As I mentioned before I have the same identical structure for
>>> phenotypes, but when I run the build the logs are really different
>>> for each one:
>>> [integrate] JDJDDEBUG::
>>> Retrieving:[phenotypeNotObserved.primaryIdentifier]
>>>    <<<<<<<------------
>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>> [integrate] JDJDDEBUG:: This is a collection
>>>      <<<<<<<<<<<<<<,-----------------
>>> [integrate] JDJDDEBUG:: =======================
>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>> [integrate] JDJDDEBUG:: xpathvalue:
>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>> [integrate] JDJDDEBUG:: =======================
>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]
>>>   <<<<<<<<<<<<<<<-----------------
>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>> [integrate] JDJDDEBUG:: fieldname=gene
>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to:
>>> (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>> [integrate] JDJDDEBUG:: N:1
>>> Am I missing any other place that might be setting this? Why the
>>> Allele.gene is being created and it’s not a collection?
>>> Thanks again
>>> Paulo
>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>
>>>> Hi Paulo,
>>>>
>>>> For a reverse reference to work, the labels have to match.
>>>>
>>>> If you have "genes" as the collection, then "genes" (not "gene"!)
>>>> has to be the value of the reverse reference to that collection.
>>>>
>>>> Does that make sense?
>>>>
>>>> Here are some docs:
>>>>
>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>
>>>> I would correct your data model, run a clean and rebuild and try the
>>>> query again.
>>>>
>>>> Cheers,
>>>> Julie
>>>>
>>>> PS. I'm CCing the dev list because I think this might help other
>>>> people too!
>>>>
>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>> Hi
>>>>> I am pretty sure I have tried with gene and genes, and it didn’t
>>>>> work. Anything else I would need to check?
>>>>> Thanks
>>>>> Paulo
>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>> Those reverse references don't match. You have a "gene" collection
>>>>>> but the reverse reference is "genes".
>>>>>>
>>>>>> Is that a typo?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>> Hi Julie
>>>>>>> We are having some problems with many-to-many relationships
>>>>>>> between a couple of sources. The Allele and Gene classes are
>>>>>>> supposed to have many items with relationships, which should be
>>>>>>> seen on query results, but we are only obtaining one gene per
>>>>>>> allele (variation).
>>>>>>> Our project addition file is on
>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>> and the Allele class has
>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>> while the Gene has
>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>> would this be correct? I have tried all possibilities in testing,
>>>>>>> and worked a lot on the mapping and how these classes and
>>>>>>> elements are loaded in the DB, with much success. A
>>>>>>> representative query would be
>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier
>>>>>>> Allele.symbol Allele.gene.primaryIdentifier
>>>>>>> Allele.gene.secondaryIdentifier Allele.gene.symbol"
>>>>>>> longDescription="" sortOrder="Allele.primaryIdentifier asc">
>>>>>>> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>> Any help appreciated
>>>>>>> Thanks
>>>>>>> Paulo
>
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

Paulo Nuin

Ciao Sergio

The CDS and Transcript are legacy and not used anymore, at least not from the mapping we have for the XML load. We have other collections in Alleles that work perfectly, like Phenotypes and Phnotypes not observed.

The mapping files are on

https://github.com/WormBase/intermine/tree/im-267/wormmine/support/properties_xpath

There might be a change here and there that were not committed but it’s basically these that we are using.

Grazie

Paulo





> On Nov 21, 2018, at 10:44 AM, sergio contrino <[hidden email]> wrote:
>
> hi paulo,
> interesting your generic approach: do your system works for CDSs and trascripts to allele (i.e. it is a problem only for the gene allele relationship)?
> could you send a link also to the mapping file(s)?
> thanks
> sergio
>
>
> On 21/11/2018 14:18, Paulo Nuin wrote:
>> Hi Julie
>> Here what an Allele item looks like (all *s are NULL)
>> -[ RECORD 1 ]---------------+-------------------------------
>> othername                   | cewivar00722050
>> linkedto                    | *
>> interpolatedmapposition     | *
>> lastupdated                 | *
>> sequencestatus              | Sequenced
>> koallele                    | *
>> nemaallele                  | *
>> natureofvariation           | *
>> rearrangement               | *
>> status                      | Live
>> intermine_method            | *
>> nbpallele                   | false
>> mutagen                     | *
>> amber_uag                   | *
>> publicname                  | *
>> type                        | Natural_variant
>> species                     | Caenorhabditis elegans
>> id                          | 1000004
>> name                        | *
>> productionmethod            | *
>> missensechange              | *
>> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->
>
>
>> secondaryidentifier         | *
>> positiveclone               | *
>> naturalvariant              | *
>> opal_uga                    | *
>> ochre_uaa                   | *
>> typeofmutation              | Deletion
>> donorsplicesite             | *
>> symbol                      | WBVar01994782
>> acceptorsplicesite          | *
>> primaryidentifier           | WBVar01994782
>> detectionmethod             | *
>> koconsortiumallele          | false
>> organismid                  | *
>> sequenceontologytermid      | 1000003
>> geneid                      | 1000005
>> class                       | org.intermine.model.bio.Allele
>> And I have
>> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>>  affectedbyalleles | affectsgenes
>> -------------------+--------------
>> (0 rows)
>> Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at
>> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>> I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection.
>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>> [integrate] JDJDDEBUG:: cast type: null
>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>> [integrate] JDJDDEBUG:: passed
>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>> [integrate] JDJDDEBUG:: false
>> I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.
>> Cheers
>> Paulo
>>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>> Paulo,
>>>
>>> (CCing the dev list for help.)
>>>
>>> Yes your data model looks correct now!
>>>
>>> What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?
>>>
>>> I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:
>>>
>>> allele.addToCollection("affectsGenes", gene)
>>> gene.addToCollection("affectedByAlleles", allele)
>>>
>>> Here are some docs on many-to-many relationships:
>>>
>>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>>
>>> Does that help?
>>> Julie
>>>
>>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>>> Hi Julie
>>>> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>>>>         <class name="Gene" is-interface="true">
>>>>                 <attribute name="operon" type="java.lang.String"/>
>>>>                 <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>>>>                 <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>>>>                 <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>>                 <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>>                 <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>>>>                 <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>>>>                 <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>>>>                 <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>>         </class>
>>>>  <class name="Allele" is-interface="true">
>>>>           <attribute name="naturalVariant" type="java.lang.String" />
>>>>           <attribute name="method" type="java.lang.String" />
>>>>           <attribute name="type" type="java.lang.String" />
>>>>           <attribute name="publicName" type="java.lang.String" />
>>>>           <attribute name="otherName" type="java.lang.String" />
>>>>           <attribute name="KOAllele" type="java.lang.String" />
>>>>           <attribute name="NBPAllele" type="java.lang.String" />
>>>>           <attribute name="nemaAllele" type="java.lang.String" />
>>>>           <attribute name="amber_UAG" type="java.lang.String" />
>>>>           <attribute name="ochre_UAA" type="java.lang.String" />
>>>>           <attribute name="opal_UGA" type="java.lang.String" />
>>>>           <attribute name="donorSpliceSite" type="java.lang.String" />
>>>>           <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>>           <attribute name="productionMethod" type="java.lang.String" />
>>>> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>>>>           <attribute name="status" type="java.lang.String" />
>>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>>           <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>>>>           <attribute name="positiveClone" type="java.lang.String" />
>>>>           <attribute name="typeOfMutation" type="java.lang.String" />
>>>>           <attribute name="sequenceStatus" type="java.lang.String" />
>>>>           <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>>           <attribute name="missenseChange" type="java.lang.String" />
>>>>           <attribute name="mutagen" type="java.lang.String" />
>>>>           <attribute name="natureOfVariation" type="java.lang.String" />
>>>>           <attribute name="interpolatedMapPosition" type="java.lang.String" />
>>>>           <attribute name="species" type="java.lang.String" />
>>>>           <attribute name="rearrangement" type="java.lang.String" />
>>>>           <attribute name="linkedTo" type="java.lang.String" />
>>>>           <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>>>>           <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
>>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>>>>           <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>>>>           <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>>           <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>>>>           <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>>>>           <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>>>>         </class>
>>>> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>>>> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>>> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
>>>> [integrate] JDJDDEBUG:: =======================
>>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>>> [integrate] JDJDDEBUG:: xpathvalue:
>>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>>> [integrate] JDJDDEBUG:: =======================
>>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>> [integrate] JDJDDEBUG:: fieldname=gene
>>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>>> [integrate] JDJDDEBUG:: N:1
>>>> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>>>> Thanks again
>>>> Paulo
>>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>
>>>>> Hi Paulo,
>>>>>
>>>>> For a reverse reference to work, the labels have to match.
>>>>>
>>>>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>>>>
>>>>> Does that make sense?
>>>>>
>>>>> Here are some docs:
>>>>>
>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>>
>>>>> I would correct your data model, run a clean and rebuild and try the query again.
>>>>>
>>>>> Cheers,
>>>>> Julie
>>>>>
>>>>> PS. I'm CCing the dev list because I think this might help other people too!
>>>>>
>>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>>> Hi
>>>>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>>>>> Thanks
>>>>>> Paulo
>>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>
>>>>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>>>>
>>>>>>> Is that a typo?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>>> Hi Julie
>>>>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>>>>> Our project addition file is on
>>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>>> and the Allele class has
>>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>>> while the Gene has
>>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>>> Any help appreciated
>>>>>>>> Thanks
>>>>>>>> Paulo
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

sergio contrino-2
dear paul,
do you have any news on this? any luck with the other debugging options?
if not we have probably to try to reproduce the issue here.
thanks
sergio

On 21/11/2018 18:58, Paulo Nuin wrote:

>
> Ciao Sergio
>
> The CDS and Transcript are legacy and not used anymore, at least not from the mapping we have for the XML load. We have other collections in Alleles that work perfectly, like Phenotypes and Phnotypes not observed.
>
> The mapping files are on
>
> https://github.com/WormBase/intermine/tree/im-267/wormmine/support/properties_xpath
>
> There might be a change here and there that were not committed but it’s basically these that we are using.
>
> Grazie
>
> Paulo
>
>
>
>
>
>> On Nov 21, 2018, at 10:44 AM, sergio contrino <[hidden email]> wrote:
>>
>> hi paulo,
>> interesting your generic approach: do your system works for CDSs and trascripts to allele (i.e. it is a problem only for the gene allele relationship)?
>> could you send a link also to the mapping file(s)?
>> thanks
>> sergio
>>
>>
>> On 21/11/2018 14:18, Paulo Nuin wrote:
>>> Hi Julie
>>> Here what an Allele item looks like (all *s are NULL)
>>> -[ RECORD 1 ]---------------+-------------------------------
>>> othername                   | cewivar00722050
>>> linkedto                    | *
>>> interpolatedmapposition     | *
>>> lastupdated                 | *
>>> sequencestatus              | Sequenced
>>> koallele                    | *
>>> nemaallele                  | *
>>> natureofvariation           | *
>>> rearrangement               | *
>>> status                      | Live
>>> intermine_method            | *
>>> nbpallele                   | false
>>> mutagen                     | *
>>> amber_uag                   | *
>>> publicname                  | *
>>> type                        | Natural_variant
>>> species                     | Caenorhabditis elegans
>>> id                          | 1000004
>>> name                        | *
>>> productionmethod            | *
>>> missensechange              | *
>>> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->
>>
>>
>>> secondaryidentifier         | *
>>> positiveclone               | *
>>> naturalvariant              | *
>>> opal_uga                    | *
>>> ochre_uaa                   | *
>>> typeofmutation              | Deletion
>>> donorsplicesite             | *
>>> symbol                      | WBVar01994782
>>> acceptorsplicesite          | *
>>> primaryidentifier           | WBVar01994782
>>> detectionmethod             | *
>>> koconsortiumallele          | false
>>> organismid                  | *
>>> sequenceontologytermid      | 1000003
>>> geneid                      | 1000005
>>> class                       | org.intermine.model.bio.Allele
>>> And I have
>>> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>>>   affectedbyalleles | affectsgenes
>>> -------------------+--------------
>>> (0 rows)
>>> Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at
>>> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>>> I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection.
>>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>>> [integrate] JDJDDEBUG:: cast type: null
>>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>>> [integrate] JDJDDEBUG:: passed
>>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>>> [integrate] JDJDDEBUG:: false
>>> I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.
>>> Cheers
>>> Paulo
>>>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>
>>>> Paulo,
>>>>
>>>> (CCing the dev list for help.)
>>>>
>>>> Yes your data model looks correct now!
>>>>
>>>> What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?
>>>>
>>>> I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:
>>>>
>>>> allele.addToCollection("affectsGenes", gene)
>>>> gene.addToCollection("affectedByAlleles", allele)
>>>>
>>>> Here are some docs on many-to-many relationships:
>>>>
>>>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>>>
>>>> Does that help?
>>>> Julie
>>>>
>>>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>>>> Hi Julie
>>>>> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>>>>>          <class name="Gene" is-interface="true">
>>>>>                  <attribute name="operon" type="java.lang.String"/>
>>>>>                  <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>>>>>                  <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>>>>>                  <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>>>                  <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>>>                  <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>>>>>                  <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>>>>>                  <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>>>>>                  <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>>>          </class>
>>>>>   <class name="Allele" is-interface="true">
>>>>>            <attribute name="naturalVariant" type="java.lang.String" />
>>>>>            <attribute name="method" type="java.lang.String" />
>>>>>            <attribute name="type" type="java.lang.String" />
>>>>>            <attribute name="publicName" type="java.lang.String" />
>>>>>            <attribute name="otherName" type="java.lang.String" />
>>>>>            <attribute name="KOAllele" type="java.lang.String" />
>>>>>            <attribute name="NBPAllele" type="java.lang.String" />
>>>>>            <attribute name="nemaAllele" type="java.lang.String" />
>>>>>            <attribute name="amber_UAG" type="java.lang.String" />
>>>>>            <attribute name="ochre_UAA" type="java.lang.String" />
>>>>>            <attribute name="opal_UGA" type="java.lang.String" />
>>>>>            <attribute name="donorSpliceSite" type="java.lang.String" />
>>>>>            <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>>>            <attribute name="productionMethod" type="java.lang.String" />
>>>>> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>>>>>            <attribute name="status" type="java.lang.String" />
>>>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>>>            <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>>>>>            <attribute name="positiveClone" type="java.lang.String" />
>>>>>            <attribute name="typeOfMutation" type="java.lang.String" />
>>>>>            <attribute name="sequenceStatus" type="java.lang.String" />
>>>>>            <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>>>            <attribute name="missenseChange" type="java.lang.String" />
>>>>>            <attribute name="mutagen" type="java.lang.String" />
>>>>>            <attribute name="natureOfVariation" type="java.lang.String" />
>>>>>            <attribute name="interpolatedMapPosition" type="java.lang.String" />
>>>>>            <attribute name="species" type="java.lang.String" />
>>>>>            <attribute name="rearrangement" type="java.lang.String" />
>>>>>            <attribute name="linkedTo" type="java.lang.String" />
>>>>>            <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>>>>>            <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
>>>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>>>>>            <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>>>>>            <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>>>            <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>>>>>            <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>>>>>            <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>>>>>          </class>
>>>>> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>>>>> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>>>> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
>>>>> [integrate] JDJDDEBUG:: =======================
>>>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>>>> [integrate] JDJDDEBUG:: xpathvalue:
>>>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>>>> [integrate] JDJDDEBUG:: =======================
>>>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>> [integrate] JDJDDEBUG:: fieldname=gene
>>>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>>>> [integrate] JDJDDEBUG:: N:1
>>>>> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>>>>> Thanks again
>>>>> Paulo
>>>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>
>>>>>> Hi Paulo,
>>>>>>
>>>>>> For a reverse reference to work, the labels have to match.
>>>>>>
>>>>>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>>>>>
>>>>>> Does that make sense?
>>>>>>
>>>>>> Here are some docs:
>>>>>>
>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>>>
>>>>>> I would correct your data model, run a clean and rebuild and try the query again.
>>>>>>
>>>>>> Cheers,
>>>>>> Julie
>>>>>>
>>>>>> PS. I'm CCing the dev list because I think this might help other people too!
>>>>>>
>>>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>>>> Hi
>>>>>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>>>>>> Thanks
>>>>>>> Paulo
>>>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>>>>>
>>>>>>>> Is that a typo?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>>>> Hi Julie
>>>>>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>>>>>> Our project addition file is on
>>>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>>>> and the Allele class has
>>>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>>>> while the Gene has
>>>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>>>> Any help appreciated
>>>>>>>>> Thanks
>>>>>>>>> Paulo
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

Paulo Nuin
Ciao Sergio

No, no luck. I have probably run about 100 test builds since we last exchanged emails. Some issues along the way:

1 - If I enter these in the model

                <collection name="alleles" referenced-type="Allele" reverse-reference="genes" />

                <collection name="genes" referenced-type="Gene" reverse-reference="alleles" />

        I get an error on build that says that one reverse reference is missing. If I change to

                <collection name=“VARIATIONS" referenced-type="Allele" reverse-reference="genes" />

                <collection name="genes" referenced-type="Gene" reverse-reference=“VARIATIONS" />

        (caps to show the difference), the build works fine. I am assuming that there is some main class that assigns Allele.gene (and maybe Gene.allele), but I don’t know where. The only mention I see for a Allele.gene is in the genomics.priorities file and in some Flymine code.


2- I made the change above and the build (for these two classes worked OK), but I have not set a mapping XPATH to load to these specific items, but I will re-run to check my debugging log.

Thanks for the help. I will post some other things about the log when I get it set.

Cheers
Paulo




       

> On Nov 27, 2018, at 4:12 AM, sergio contrino <[hidden email]> wrote:
>
> dear paul,
> do you have any news on this? any luck with the other debugging options?
> if not we have probably to try to reproduce the issue here.
> thanks
> sergio
>
> On 21/11/2018 18:58, Paulo Nuin wrote:
>> Ciao Sergio
>> The CDS and Transcript are legacy and not used anymore, at least not from the mapping we have for the XML load. We have other collections in Alleles that work perfectly, like Phenotypes and Phnotypes not observed.
>> The mapping files are on
>> https://github.com/WormBase/intermine/tree/im-267/wormmine/support/properties_xpath
>> There might be a change here and there that were not committed but it’s basically these that we are using.
>> Grazie
>> Paulo
>>> On Nov 21, 2018, at 10:44 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> hi paulo,
>>> interesting your generic approach: do your system works for CDSs and trascripts to allele (i.e. it is a problem only for the gene allele relationship)?
>>> could you send a link also to the mapping file(s)?
>>> thanks
>>> sergio
>>>
>>>
>>> On 21/11/2018 14:18, Paulo Nuin wrote:
>>>> Hi Julie
>>>> Here what an Allele item looks like (all *s are NULL)
>>>> -[ RECORD 1 ]---------------+-------------------------------
>>>> othername                   | cewivar00722050
>>>> linkedto                    | *
>>>> interpolatedmapposition     | *
>>>> lastupdated                 | *
>>>> sequencestatus              | Sequenced
>>>> koallele                    | *
>>>> nemaallele                  | *
>>>> natureofvariation           | *
>>>> rearrangement               | *
>>>> status                      | Live
>>>> intermine_method            | *
>>>> nbpallele                   | false
>>>> mutagen                     | *
>>>> amber_uag                   | *
>>>> publicname                  | *
>>>> type                        | Natural_variant
>>>> species                     | Caenorhabditis elegans
>>>> id                          | 1000004
>>>> name                        | *
>>>> productionmethod            | *
>>>> missensechange              | *
>>>> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->
>>>
>>>
>>>> secondaryidentifier         | *
>>>> positiveclone               | *
>>>> naturalvariant              | *
>>>> opal_uga                    | *
>>>> ochre_uaa                   | *
>>>> typeofmutation              | Deletion
>>>> donorsplicesite             | *
>>>> symbol                      | WBVar01994782
>>>> acceptorsplicesite          | *
>>>> primaryidentifier           | WBVar01994782
>>>> detectionmethod             | *
>>>> koconsortiumallele          | false
>>>> organismid                  | *
>>>> sequenceontologytermid      | 1000003
>>>> geneid                      | 1000005
>>>> class                       | org.intermine.model.bio.Allele
>>>> And I have
>>>> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>>>>  affectedbyalleles | affectsgenes
>>>> -------------------+--------------
>>>> (0 rows)
>>>> Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at
>>>> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>>>> I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection.
>>>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>>>> [integrate] JDJDDEBUG:: cast type: null
>>>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>>>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>>>> [integrate] JDJDDEBUG:: passed
>>>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>>>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>>>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>>>> [integrate] JDJDDEBUG:: false
>>>> I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.
>>>> Cheers
>>>> Paulo
>>>>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>
>>>>> Paulo,
>>>>>
>>>>> (CCing the dev list for help.)
>>>>>
>>>>> Yes your data model looks correct now!
>>>>>
>>>>> What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?
>>>>>
>>>>> I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:
>>>>>
>>>>> allele.addToCollection("affectsGenes", gene)
>>>>> gene.addToCollection("affectedByAlleles", allele)
>>>>>
>>>>> Here are some docs on many-to-many relationships:
>>>>>
>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>>>>
>>>>> Does that help?
>>>>> Julie
>>>>>
>>>>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>>>>> Hi Julie
>>>>>> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>>>>>>         <class name="Gene" is-interface="true">
>>>>>>                 <attribute name="operon" type="java.lang.String"/>
>>>>>>                 <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>>>>>>                 <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>>>>>>                 <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>>>>                 <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>>>>                 <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>>>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>>>>>>                 <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>>>>>>                 <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>>>>>>                 <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>>>>         </class>
>>>>>>  <class name="Allele" is-interface="true">
>>>>>>           <attribute name="naturalVariant" type="java.lang.String" />
>>>>>>           <attribute name="method" type="java.lang.String" />
>>>>>>           <attribute name="type" type="java.lang.String" />
>>>>>>           <attribute name="publicName" type="java.lang.String" />
>>>>>>           <attribute name="otherName" type="java.lang.String" />
>>>>>>           <attribute name="KOAllele" type="java.lang.String" />
>>>>>>           <attribute name="NBPAllele" type="java.lang.String" />
>>>>>>           <attribute name="nemaAllele" type="java.lang.String" />
>>>>>>           <attribute name="amber_UAG" type="java.lang.String" />
>>>>>>           <attribute name="ochre_UAA" type="java.lang.String" />
>>>>>>           <attribute name="opal_UGA" type="java.lang.String" />
>>>>>>           <attribute name="donorSpliceSite" type="java.lang.String" />
>>>>>>           <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>>>>           <attribute name="productionMethod" type="java.lang.String" />
>>>>>> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>>>>>>           <attribute name="status" type="java.lang.String" />
>>>>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>>>>           <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>>>>>>           <attribute name="positiveClone" type="java.lang.String" />
>>>>>>           <attribute name="typeOfMutation" type="java.lang.String" />
>>>>>>           <attribute name="sequenceStatus" type="java.lang.String" />
>>>>>>           <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>>>>           <attribute name="missenseChange" type="java.lang.String" />
>>>>>>           <attribute name="mutagen" type="java.lang.String" />
>>>>>>           <attribute name="natureOfVariation" type="java.lang.String" />
>>>>>>           <attribute name="interpolatedMapPosition" type="java.lang.String" />
>>>>>>           <attribute name="species" type="java.lang.String" />
>>>>>>           <attribute name="rearrangement" type="java.lang.String" />
>>>>>>           <attribute name="linkedTo" type="java.lang.String" />
>>>>>>           <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>>>>>>           <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
>>>>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>>>>>>           <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>>>>>>           <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>>>>           <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>>>>>>           <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>>>>>>           <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>>>>>>         </class>
>>>>>> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>>>>>> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>>>>> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>>>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>>>>> [integrate] JDJDDEBUG:: xpathvalue:
>>>>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>> [integrate] JDJDDEBUG:: fieldname=gene
>>>>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>>>>> [integrate] JDJDDEBUG:: N:1
>>>>>> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>>>>>> Thanks again
>>>>>> Paulo
>>>>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi Paulo,
>>>>>>>
>>>>>>> For a reverse reference to work, the labels have to match.
>>>>>>>
>>>>>>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>>>>>>
>>>>>>> Does that make sense?
>>>>>>>
>>>>>>> Here are some docs:
>>>>>>>
>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>>>>
>>>>>>> I would correct your data model, run a clean and rebuild and try the query again.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Julie
>>>>>>>
>>>>>>> PS. I'm CCing the dev list because I think this might help other people too!
>>>>>>>
>>>>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>>>>> Hi
>>>>>>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>>>>>>> Thanks
>>>>>>>> Paulo
>>>>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>>>>>>
>>>>>>>>> Is that a typo?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>>>>> Hi Julie
>>>>>>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>>>>>>> Our project addition file is on
>>>>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>>>>> and the Allele class has
>>>>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>>>>> while the Gene has
>>>>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>>>>> Any help appreciated
>>>>>>>>>> Thanks
>>>>>>>>>> Paulo
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

Paulo Nuin
Hi again

So, I did a quick build of both classes and this is what I see from the log:

Loading Gene

[integrate] JDJDDEBUG:: Retrieving:[primaryIdentifier]
[integrate] JDJDDEBUG:: fieldname=primaryIdentifier
[integrate] JDJDDEBUG:: xpathvalue:WBGene00000898
[integrate] JDJDDEBUG:: Retrieving:[alleles.primaryIdentifier]
[integrate] JDJDDEBUG:: suffix:primaryIdentifier
[integrate] JDJDDEBUG:: fieldname=alleles
[integrate] JDJDDEBUG:: This is a collection <-------------------------
[integrate] JDJDDEBUG:: new Allele object:WBVar00143694
[integrate] JDJDDEBUG:: alleles:[WBVar00143694]
[integrate] JDJDDEBUG:: new Allele object:WBVar00143948
[integrate] JDJDDEBUG:: alleles:[WBVar00143948]
[integrate] JDJDDEBUG:: new Allele object:WBVar00143891
[integrate] JDJDDEBUG:: alleles:[WBVar00143891]
[integrate] JDJDDEBUG:: new Allele object:WBVar00143945
[integrate] JDJDDEBUG:: alleles:[WBVar00143945]
[integrate] JDJDDEBUG:: new Allele object:WBVar00143947
[integrate] JDJDDEBUG:: alleles:[WBVar00143947]
[integrate] JDJDDEBUG:: new Allele object:WBVar00143949
[integrate] JDJDDEBUG:: alleles:[WBVar00143949]
[integrate] JDJDDEBUG:: new Allele object:WBVar00143967
[integrate] JDJDDEBUG:: alleles:[WBVar00143967]
[integrate] JDJDDEBUG:: new Allele object:WBVar00088550
[integrate] JDJDDEBUG:: alleles:[WBVar00088550]
[integrate] JDJDDEBUG:: new Allele object:WBVar00088710
[integrate] JDJDDEBUG:: alleles:[WBVar00088710]
[integrate] JDJDDEBUG:: new Allele object:WBVar00088714
[integrate] JDJDDEBUG:: alleles:[WBVar00088714]
[integrate] JDJDDEBUG:: new Allele object:WBVar00088892
[integrate] JDJDDEBUG:: alleles:[WBVar00088892]
[integrate] JDJDDEBUG:: new Allele object:WBVar00242507
[integrate] JDJDDEBUG:: alleles:[WBVar00242507]
[integrate] JDJDDEBUG:: new Allele object:WBVar00242509
[integrate] JDJDDEBUG:: alleles:[WBVar00242509]
[integrate] JDJDDEBUG:: new Allele object:WBVar00242513
[integrate] JDJDDEBUG:: alleles:[WBVar00242513]
[integrate] JDJDDEBUG:: new Allele object:WBVar00242527
[integrate] JDJDDEBUG:: alleles:[WBVar00242527]
[integrate] JDJDDEBUG:: new Allele object:WBVar00242530
[integrate] JDJDDEBUG:: alleles:[WBVar00242530]

It seems to get the Allele items properly.


Loading Allele

[integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
[integrate] JDJDDEBUG:: cast type: null
[integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
[integrate] JDJDDEBUG:: gene.primaryIdentifier
[integrate] JDJDDEBUG:: passed
[integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
[integrate] JDJDDEBUG:: new1 primaryIdentifier
[integrate] JDJDDEBUG:: new2 primaryIdentifier
[integrate] JDJDDEBUG:: false
…..

[integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]
[integrate] JDJDDEBUG:: suffix:primaryIdentifier
[integrate] JDJDDEBUG:: fieldname=gene
[integrate] JDJDDEBUG:: new Gene object:WBGene00000011
[integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000011]
[integrate] JDJDDEBUG:: N:1

It seems that it still doesn’t see it as a collection.

But in the end I don’t see where the relationship ends being created. The Gene and Allele table items are bare and the relations table are empty

intermine_dev_268_small=# select * from genesvariations;
 variations | genes
------------+-------
(0 rows)

Cheers
Paulo






> On Nov 27, 2018, at 8:37 AM, Paulo Nuin <[hidden email]> wrote:
>
> Ciao Sergio
>
> No, no luck. I have probably run about 100 test builds since we last exchanged emails. Some issues along the way:
>
> 1 - If I enter these in the model
>
>                <collection name="alleles" referenced-type="Allele" reverse-reference="genes" />
>
> <collection name="genes" referenced-type="Gene" reverse-reference="alleles" />
>
> I get an error on build that says that one reverse reference is missing. If I change to
>
> <collection name=“VARIATIONS" referenced-type="Allele" reverse-reference="genes" />
>
> <collection name="genes" referenced-type="Gene" reverse-reference=“VARIATIONS" />
>
> (caps to show the difference), the build works fine. I am assuming that there is some main class that assigns Allele.gene (and maybe Gene.allele), but I don’t know where. The only mention I see for a Allele.gene is in the genomics.priorities file and in some Flymine code.
>
>
> 2- I made the change above and the build (for these two classes worked OK), but I have not set a mapping XPATH to load to these specific items, but I will re-run to check my debugging log.
>
> Thanks for the help. I will post some other things about the log when I get it set.
>
> Cheers
> Paulo
>
>
>
>
>
>
>> On Nov 27, 2018, at 4:12 AM, sergio contrino <[hidden email]> wrote:
>>
>> dear paul,
>> do you have any news on this? any luck with the other debugging options?
>> if not we have probably to try to reproduce the issue here.
>> thanks
>> sergio
>>
>> On 21/11/2018 18:58, Paulo Nuin wrote:
>>> Ciao Sergio
>>> The CDS and Transcript are legacy and not used anymore, at least not from the mapping we have for the XML load. We have other collections in Alleles that work perfectly, like Phenotypes and Phnotypes not observed.
>>> The mapping files are on
>>> https://github.com/WormBase/intermine/tree/im-267/wormmine/support/properties_xpath
>>> There might be a change here and there that were not committed but it’s basically these that we are using.
>>> Grazie
>>> Paulo
>>>> On Nov 21, 2018, at 10:44 AM, sergio contrino <[hidden email]> wrote:
>>>>
>>>> hi paulo,
>>>> interesting your generic approach: do your system works for CDSs and trascripts to allele (i.e. it is a problem only for the gene allele relationship)?
>>>> could you send a link also to the mapping file(s)?
>>>> thanks
>>>> sergio
>>>>
>>>>
>>>> On 21/11/2018 14:18, Paulo Nuin wrote:
>>>>> Hi Julie
>>>>> Here what an Allele item looks like (all *s are NULL)
>>>>> -[ RECORD 1 ]---------------+-------------------------------
>>>>> othername                   | cewivar00722050
>>>>> linkedto                    | *
>>>>> interpolatedmapposition     | *
>>>>> lastupdated                 | *
>>>>> sequencestatus              | Sequenced
>>>>> koallele                    | *
>>>>> nemaallele                  | *
>>>>> natureofvariation           | *
>>>>> rearrangement               | *
>>>>> status                      | Live
>>>>> intermine_method            | *
>>>>> nbpallele                   | false
>>>>> mutagen                     | *
>>>>> amber_uag                   | *
>>>>> publicname                  | *
>>>>> type                        | Natural_variant
>>>>> species                     | Caenorhabditis elegans
>>>>> id                          | 1000004
>>>>> name                        | *
>>>>> productionmethod            | *
>>>>> missensechange              | *
>>>>> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->
>>>>
>>>>
>>>>> secondaryidentifier         | *
>>>>> positiveclone               | *
>>>>> naturalvariant              | *
>>>>> opal_uga                    | *
>>>>> ochre_uaa                   | *
>>>>> typeofmutation              | Deletion
>>>>> donorsplicesite             | *
>>>>> symbol                      | WBVar01994782
>>>>> acceptorsplicesite          | *
>>>>> primaryidentifier           | WBVar01994782
>>>>> detectionmethod             | *
>>>>> koconsortiumallele          | false
>>>>> organismid                  | *
>>>>> sequenceontologytermid      | 1000003
>>>>> geneid                      | 1000005
>>>>> class                       | org.intermine.model.bio.Allele
>>>>> And I have
>>>>> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>>>>> affectedbyalleles | affectsgenes
>>>>> -------------------+--------------
>>>>> (0 rows)
>>>>> Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at
>>>>> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>>>>> I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection.
>>>>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>>>>> [integrate] JDJDDEBUG:: cast type: null
>>>>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>>>>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>>>>> [integrate] JDJDDEBUG:: passed
>>>>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>>>>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>>>>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>>>>> [integrate] JDJDDEBUG:: false
>>>>> I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.
>>>>> Cheers
>>>>> Paulo
>>>>>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>
>>>>>> Paulo,
>>>>>>
>>>>>> (CCing the dev list for help.)
>>>>>>
>>>>>> Yes your data model looks correct now!
>>>>>>
>>>>>> What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?
>>>>>>
>>>>>> I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:
>>>>>>
>>>>>> allele.addToCollection("affectsGenes", gene)
>>>>>> gene.addToCollection("affectedByAlleles", allele)
>>>>>>
>>>>>> Here are some docs on many-to-many relationships:
>>>>>>
>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>>>>>
>>>>>> Does that help?
>>>>>> Julie
>>>>>>
>>>>>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>>>>>> Hi Julie
>>>>>>> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>>>>>>>        <class name="Gene" is-interface="true">
>>>>>>>                <attribute name="operon" type="java.lang.String"/>
>>>>>>>                <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>>>>>>>                <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>>>>>>>                <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>>>>>                <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>>>>>                <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>>>>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>>>>>>>                <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>>>>>>>                <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>>>>>>>                <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>>>>>        </class>
>>>>>>> <class name="Allele" is-interface="true">
>>>>>>>          <attribute name="naturalVariant" type="java.lang.String" />
>>>>>>>          <attribute name="method" type="java.lang.String" />
>>>>>>>          <attribute name="type" type="java.lang.String" />
>>>>>>>          <attribute name="publicName" type="java.lang.String" />
>>>>>>>          <attribute name="otherName" type="java.lang.String" />
>>>>>>>          <attribute name="KOAllele" type="java.lang.String" />
>>>>>>>          <attribute name="NBPAllele" type="java.lang.String" />
>>>>>>>          <attribute name="nemaAllele" type="java.lang.String" />
>>>>>>>          <attribute name="amber_UAG" type="java.lang.String" />
>>>>>>>          <attribute name="ochre_UAA" type="java.lang.String" />
>>>>>>>          <attribute name="opal_UGA" type="java.lang.String" />
>>>>>>>          <attribute name="donorSpliceSite" type="java.lang.String" />
>>>>>>>          <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>>>>>          <attribute name="productionMethod" type="java.lang.String" />
>>>>>>> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>>>>>>>          <attribute name="status" type="java.lang.String" />
>>>>>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>>>>>          <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>>>>>>>          <attribute name="positiveClone" type="java.lang.String" />
>>>>>>>          <attribute name="typeOfMutation" type="java.lang.String" />
>>>>>>>          <attribute name="sequenceStatus" type="java.lang.String" />
>>>>>>>          <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>>>>>          <attribute name="missenseChange" type="java.lang.String" />
>>>>>>>          <attribute name="mutagen" type="java.lang.String" />
>>>>>>>          <attribute name="natureOfVariation" type="java.lang.String" />
>>>>>>>          <attribute name="interpolatedMapPosition" type="java.lang.String" />
>>>>>>>          <attribute name="species" type="java.lang.String" />
>>>>>>>          <attribute name="rearrangement" type="java.lang.String" />
>>>>>>>          <attribute name="linkedTo" type="java.lang.String" />
>>>>>>>          <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>>>>>>>          <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
>>>>>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>>>>>>>          <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>>>>>>>          <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>>>>>          <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>>>>>>>          <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>>>>>>>          <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>>>>>>>        </class>
>>>>>>> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>>>>>> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>>>>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>>>>>> [integrate] JDJDDEBUG:: xpathvalue:
>>>>>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>> [integrate] JDJDDEBUG:: fieldname=gene
>>>>>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>>>>>> [integrate] JDJDDEBUG:: N:1
>>>>>>> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>>>>>>> Thanks again
>>>>>>> Paulo
>>>>>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> Hi Paulo,
>>>>>>>>
>>>>>>>> For a reverse reference to work, the labels have to match.
>>>>>>>>
>>>>>>>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>>>>>>>
>>>>>>>> Does that make sense?
>>>>>>>>
>>>>>>>> Here are some docs:
>>>>>>>>
>>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>>>>>
>>>>>>>> I would correct your data model, run a clean and rebuild and try the query again.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Julie
>>>>>>>>
>>>>>>>> PS. I'm CCing the dev list because I think this might help other people too!
>>>>>>>>
>>>>>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>>>>>> Hi
>>>>>>>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>>>>>>>> Thanks
>>>>>>>>> Paulo
>>>>>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>>>>>>>
>>>>>>>>>> Is that a typo?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>>>>>> Hi Julie
>>>>>>>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>>>>>>>> Our project addition file is on
>>>>>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>>>>>> and the Allele class has
>>>>>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>>>>>> while the Gene has
>>>>>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>>>>>> Any help appreciated
>>>>>>>>>>> Thanks
>>>>>>>>>>> Paulo
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> [hidden email]
>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>
>>>> --
>>>> sergio contrino                  InterMine, University of Cambridge
>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

sergio contrino-2
thank you paul,
i think you are right, it seems it is considering this a reference
notwithstanding your additions xml file. i'll check your checkout (which
i assume is the im-267 branch).
thanks
sergio

On 27/11/2018 16:46, Paulo Nuin wrote:

> Hi again
>
> So, I did a quick build of both classes and this is what I see from the log:
>
> Loading Gene
>
> [integrate] JDJDDEBUG:: Retrieving:[primaryIdentifier]
> [integrate] JDJDDEBUG:: fieldname=primaryIdentifier
> [integrate] JDJDDEBUG:: xpathvalue:WBGene00000898
> [integrate] JDJDDEBUG:: Retrieving:[alleles.primaryIdentifier]
> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
> [integrate] JDJDDEBUG:: fieldname=alleles
> [integrate] JDJDDEBUG:: This is a collection <-------------------------
> [integrate] JDJDDEBUG:: new Allele object:WBVar00143694
> [integrate] JDJDDEBUG:: alleles:[WBVar00143694]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00143948
> [integrate] JDJDDEBUG:: alleles:[WBVar00143948]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00143891
> [integrate] JDJDDEBUG:: alleles:[WBVar00143891]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00143945
> [integrate] JDJDDEBUG:: alleles:[WBVar00143945]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00143947
> [integrate] JDJDDEBUG:: alleles:[WBVar00143947]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00143949
> [integrate] JDJDDEBUG:: alleles:[WBVar00143949]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00143967
> [integrate] JDJDDEBUG:: alleles:[WBVar00143967]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00088550
> [integrate] JDJDDEBUG:: alleles:[WBVar00088550]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00088710
> [integrate] JDJDDEBUG:: alleles:[WBVar00088710]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00088714
> [integrate] JDJDDEBUG:: alleles:[WBVar00088714]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00088892
> [integrate] JDJDDEBUG:: alleles:[WBVar00088892]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00242507
> [integrate] JDJDDEBUG:: alleles:[WBVar00242507]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00242509
> [integrate] JDJDDEBUG:: alleles:[WBVar00242509]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00242513
> [integrate] JDJDDEBUG:: alleles:[WBVar00242513]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00242527
> [integrate] JDJDDEBUG:: alleles:[WBVar00242527]
> [integrate] JDJDDEBUG:: new Allele object:WBVar00242530
> [integrate] JDJDDEBUG:: alleles:[WBVar00242530]
>
> It seems to get the Allele items properly.
>
>
> Loading Allele
>
> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
> [integrate] JDJDDEBUG:: cast type: null
> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
> [integrate] JDJDDEBUG:: gene.primaryIdentifier
> [integrate] JDJDDEBUG:: passed
> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
> [integrate] JDJDDEBUG:: new1 primaryIdentifier
> [integrate] JDJDDEBUG:: new2 primaryIdentifier
> [integrate] JDJDDEBUG:: false
> …..
>
> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]
> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
> [integrate] JDJDDEBUG:: fieldname=gene
> [integrate] JDJDDEBUG:: new Gene object:WBGene00000011
> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000011]
> [integrate] JDJDDEBUG:: N:1
>
> It seems that it still doesn’t see it as a collection.
>
> But in the end I don’t see where the relationship ends being created. The Gene and Allele table items are bare and the relations table are empty
>
> intermine_dev_268_small=# select * from genesvariations;
>   variations | genes
> ------------+-------
> (0 rows)
>
> Cheers
> Paulo
>
>
>
>
>
>
>> On Nov 27, 2018, at 8:37 AM, Paulo Nuin <[hidden email]> wrote:
>>
>> Ciao Sergio
>>
>> No, no luck. I have probably run about 100 test builds since we last exchanged emails. Some issues along the way:
>>
>> 1 - If I enter these in the model
>>
>>                 <collection name="alleles" referenced-type="Allele" reverse-reference="genes" />
>>
>> <collection name="genes" referenced-type="Gene" reverse-reference="alleles" />
>>
>> I get an error on build that says that one reverse reference is missing. If I change to
>>
>> <collection name=“VARIATIONS" referenced-type="Allele" reverse-reference="genes" />
>>
>> <collection name="genes" referenced-type="Gene" reverse-reference=“VARIATIONS" />
>>
>> (caps to show the difference), the build works fine. I am assuming that there is some main class that assigns Allele.gene (and maybe Gene.allele), but I don’t know where. The only mention I see for a Allele.gene is in the genomics.priorities file and in some Flymine code.
>>
>>
>> 2- I made the change above and the build (for these two classes worked OK), but I have not set a mapping XPATH to load to these specific items, but I will re-run to check my debugging log.
>>
>> Thanks for the help. I will post some other things about the log when I get it set.
>>
>> Cheers
>> Paulo
>>
>>
>>
>>
>>
>>
>>> On Nov 27, 2018, at 4:12 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> dear paul,
>>> do you have any news on this? any luck with the other debugging options?
>>> if not we have probably to try to reproduce the issue here.
>>> thanks
>>> sergio
>>>
>>> On 21/11/2018 18:58, Paulo Nuin wrote:
>>>> Ciao Sergio
>>>> The CDS and Transcript are legacy and not used anymore, at least not from the mapping we have for the XML load. We have other collections in Alleles that work perfectly, like Phenotypes and Phnotypes not observed.
>>>> The mapping files are on
>>>> https://github.com/WormBase/intermine/tree/im-267/wormmine/support/properties_xpath
>>>> There might be a change here and there that were not committed but it’s basically these that we are using.
>>>> Grazie
>>>> Paulo
>>>>> On Nov 21, 2018, at 10:44 AM, sergio contrino <[hidden email]> wrote:
>>>>>
>>>>> hi paulo,
>>>>> interesting your generic approach: do your system works for CDSs and trascripts to allele (i.e. it is a problem only for the gene allele relationship)?
>>>>> could you send a link also to the mapping file(s)?
>>>>> thanks
>>>>> sergio
>>>>>
>>>>>
>>>>> On 21/11/2018 14:18, Paulo Nuin wrote:
>>>>>> Hi Julie
>>>>>> Here what an Allele item looks like (all *s are NULL)
>>>>>> -[ RECORD 1 ]---------------+-------------------------------
>>>>>> othername                   | cewivar00722050
>>>>>> linkedto                    | *
>>>>>> interpolatedmapposition     | *
>>>>>> lastupdated                 | *
>>>>>> sequencestatus              | Sequenced
>>>>>> koallele                    | *
>>>>>> nemaallele                  | *
>>>>>> natureofvariation           | *
>>>>>> rearrangement               | *
>>>>>> status                      | Live
>>>>>> intermine_method            | *
>>>>>> nbpallele                   | false
>>>>>> mutagen                     | *
>>>>>> amber_uag                   | *
>>>>>> publicname                  | *
>>>>>> type                        | Natural_variant
>>>>>> species                     | Caenorhabditis elegans
>>>>>> id                          | 1000004
>>>>>> name                        | *
>>>>>> productionmethod            | *
>>>>>> missensechange              | *
>>>>>> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->
>>>>>
>>>>>
>>>>>> secondaryidentifier         | *
>>>>>> positiveclone               | *
>>>>>> naturalvariant              | *
>>>>>> opal_uga                    | *
>>>>>> ochre_uaa                   | *
>>>>>> typeofmutation              | Deletion
>>>>>> donorsplicesite             | *
>>>>>> symbol                      | WBVar01994782
>>>>>> acceptorsplicesite          | *
>>>>>> primaryidentifier           | WBVar01994782
>>>>>> detectionmethod             | *
>>>>>> koconsortiumallele          | false
>>>>>> organismid                  | *
>>>>>> sequenceontologytermid      | 1000003
>>>>>> geneid                      | 1000005
>>>>>> class                       | org.intermine.model.bio.Allele
>>>>>> And I have
>>>>>> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>>>>>> affectedbyalleles | affectsgenes
>>>>>> -------------------+--------------
>>>>>> (0 rows)
>>>>>> Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at
>>>>>> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>>>>>> I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection.
>>>>>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>>>>>> [integrate] JDJDDEBUG:: cast type: null
>>>>>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>>>>>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>>>>>> [integrate] JDJDDEBUG:: passed
>>>>>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>>>>>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>>>>>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>>>>>> [integrate] JDJDDEBUG:: false
>>>>>> I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.
>>>>>> Cheers
>>>>>> Paulo
>>>>>>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>
>>>>>>> Paulo,
>>>>>>>
>>>>>>> (CCing the dev list for help.)
>>>>>>>
>>>>>>> Yes your data model looks correct now!
>>>>>>>
>>>>>>> What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?
>>>>>>>
>>>>>>> I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:
>>>>>>>
>>>>>>> allele.addToCollection("affectsGenes", gene)
>>>>>>> gene.addToCollection("affectedByAlleles", allele)
>>>>>>>
>>>>>>> Here are some docs on many-to-many relationships:
>>>>>>>
>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>>>>>>
>>>>>>> Does that help?
>>>>>>> Julie
>>>>>>>
>>>>>>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>>>>>>> Hi Julie
>>>>>>>> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>>>>>>>>         <class name="Gene" is-interface="true">
>>>>>>>>                 <attribute name="operon" type="java.lang.String"/>
>>>>>>>>                 <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>>>>>>>>                 <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>>>>>>>>                 <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>>>>>>                 <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>>>>>>                 <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>>>>>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>>>>>>>>                 <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>>>>>>>>                 <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>>>>>>>>                 <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>>>>>>         </class>
>>>>>>>> <class name="Allele" is-interface="true">
>>>>>>>>           <attribute name="naturalVariant" type="java.lang.String" />
>>>>>>>>           <attribute name="method" type="java.lang.String" />
>>>>>>>>           <attribute name="type" type="java.lang.String" />
>>>>>>>>           <attribute name="publicName" type="java.lang.String" />
>>>>>>>>           <attribute name="otherName" type="java.lang.String" />
>>>>>>>>           <attribute name="KOAllele" type="java.lang.String" />
>>>>>>>>           <attribute name="NBPAllele" type="java.lang.String" />
>>>>>>>>           <attribute name="nemaAllele" type="java.lang.String" />
>>>>>>>>           <attribute name="amber_UAG" type="java.lang.String" />
>>>>>>>>           <attribute name="ochre_UAA" type="java.lang.String" />
>>>>>>>>           <attribute name="opal_UGA" type="java.lang.String" />
>>>>>>>>           <attribute name="donorSpliceSite" type="java.lang.String" />
>>>>>>>>           <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>>>>>>           <attribute name="productionMethod" type="java.lang.String" />
>>>>>>>> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>>>>>>>>           <attribute name="status" type="java.lang.String" />
>>>>>>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>>>>>>           <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>>>>>>>>           <attribute name="positiveClone" type="java.lang.String" />
>>>>>>>>           <attribute name="typeOfMutation" type="java.lang.String" />
>>>>>>>>           <attribute name="sequenceStatus" type="java.lang.String" />
>>>>>>>>           <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>>>>>>           <attribute name="missenseChange" type="java.lang.String" />
>>>>>>>>           <attribute name="mutagen" type="java.lang.String" />
>>>>>>>>           <attribute name="natureOfVariation" type="java.lang.String" />
>>>>>>>>           <attribute name="interpolatedMapPosition" type="java.lang.String" />
>>>>>>>>           <attribute name="species" type="java.lang.String" />
>>>>>>>>           <attribute name="rearrangement" type="java.lang.String" />
>>>>>>>>           <attribute name="linkedTo" type="java.lang.String" />
>>>>>>>>           <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>>>>>>>>           <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
>>>>>>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>>>>>>>>           <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>>>>>>>>           <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>>>>>>           <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>>>>>>>>           <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>>>>>>>>           <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>>>>>>>>         </class>
>>>>>>>> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
>>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>>>>>>> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
>>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>>>>>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>>>>>>> [integrate] JDJDDEBUG:: xpathvalue:
>>>>>>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
>>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>>> [integrate] JDJDDEBUG:: fieldname=gene
>>>>>>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>>>>>>> [integrate] JDJDDEBUG:: N:1
>>>>>>>> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>>>>>>>> Thanks again
>>>>>>>> Paulo
>>>>>>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Hi Paulo,
>>>>>>>>>
>>>>>>>>> For a reverse reference to work, the labels have to match.
>>>>>>>>>
>>>>>>>>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>>>>>>>>
>>>>>>>>> Does that make sense?
>>>>>>>>>
>>>>>>>>> Here are some docs:
>>>>>>>>>
>>>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>>>>>>
>>>>>>>>> I would correct your data model, run a clean and rebuild and try the query again.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Julie
>>>>>>>>>
>>>>>>>>> PS. I'm CCing the dev list because I think this might help other people too!
>>>>>>>>>
>>>>>>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>>>>>>> Hi
>>>>>>>>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>>>>>>>>> Thanks
>>>>>>>>>> Paulo
>>>>>>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>>>>>>>>
>>>>>>>>>>> Is that a typo?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>>>>>>> Hi Julie
>>>>>>>>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>>>>>>>>> Our project addition file is on
>>>>>>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>>>>>>> and the Allele class has
>>>>>>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>>>>>>> while the Gene has
>>>>>>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>>>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>>>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>>>>>>> Any help appreciated
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Paulo
>>>>>> _______________________________________________
>>>>>> dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>
>>>>> --
>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org
>>
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

Paulo Nuin
Hi

Yes, 267 is the latest, the only difference there would be the genomic additions file that I have changed in the last weeks

Thanks

Paulo



> On Nov 27, 2018, at 11:44 AM, sergio contrino <[hidden email]> wrote:
>
> thank you paul,
> i think you are right, it seems it is considering this a reference notwithstanding your additions xml file. i'll check your checkout (which i assume is the im-267 branch).
> thanks
> sergio
>
> On 27/11/2018 16:46, Paulo Nuin wrote:
>> Hi again
>> So, I did a quick build of both classes and this is what I see from the log:
>> Loading Gene
>> [integrate] JDJDDEBUG:: Retrieving:[primaryIdentifier]
>> [integrate] JDJDDEBUG:: fieldname=primaryIdentifier
>> [integrate] JDJDDEBUG:: xpathvalue:WBGene00000898
>> [integrate] JDJDDEBUG:: Retrieving:[alleles.primaryIdentifier]
>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>> [integrate] JDJDDEBUG:: fieldname=alleles
>> [integrate] JDJDDEBUG:: This is a collection <-------------------------
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143694
>> [integrate] JDJDDEBUG:: alleles:[WBVar00143694]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143948
>> [integrate] JDJDDEBUG:: alleles:[WBVar00143948]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143891
>> [integrate] JDJDDEBUG:: alleles:[WBVar00143891]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143945
>> [integrate] JDJDDEBUG:: alleles:[WBVar00143945]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143947
>> [integrate] JDJDDEBUG:: alleles:[WBVar00143947]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143949
>> [integrate] JDJDDEBUG:: alleles:[WBVar00143949]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143967
>> [integrate] JDJDDEBUG:: alleles:[WBVar00143967]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088550
>> [integrate] JDJDDEBUG:: alleles:[WBVar00088550]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088710
>> [integrate] JDJDDEBUG:: alleles:[WBVar00088710]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088714
>> [integrate] JDJDDEBUG:: alleles:[WBVar00088714]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088892
>> [integrate] JDJDDEBUG:: alleles:[WBVar00088892]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242507
>> [integrate] JDJDDEBUG:: alleles:[WBVar00242507]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242509
>> [integrate] JDJDDEBUG:: alleles:[WBVar00242509]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242513
>> [integrate] JDJDDEBUG:: alleles:[WBVar00242513]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242527
>> [integrate] JDJDDEBUG:: alleles:[WBVar00242527]
>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242530
>> [integrate] JDJDDEBUG:: alleles:[WBVar00242530]
>> It seems to get the Allele items properly.
>> Loading Allele
>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>> [integrate] JDJDDEBUG:: cast type: null
>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>> [integrate] JDJDDEBUG:: passed
>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>> [integrate] JDJDDEBUG:: false
>> …..
>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]
>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>> [integrate] JDJDDEBUG:: fieldname=gene
>> [integrate] JDJDDEBUG:: new Gene object:WBGene00000011
>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000011]
>> [integrate] JDJDDEBUG:: N:1
>> It seems that it still doesn’t see it as a collection.
>> But in the end I don’t see where the relationship ends being created. The Gene and Allele table items are bare and the relations table are empty
>> intermine_dev_268_small=# select * from genesvariations;
>>  variations | genes
>> ------------+-------
>> (0 rows)
>> Cheers
>> Paulo
>>> On Nov 27, 2018, at 8:37 AM, Paulo Nuin <[hidden email]> wrote:
>>>
>>> Ciao Sergio
>>>
>>> No, no luck. I have probably run about 100 test builds since we last exchanged emails. Some issues along the way:
>>>
>>> 1 - If I enter these in the model
>>>
>>>                <collection name="alleles" referenced-type="Allele" reverse-reference="genes" />
>>>
>>> <collection name="genes" referenced-type="Gene" reverse-reference="alleles" />
>>>
>>> I get an error on build that says that one reverse reference is missing. If I change to
>>>
>>> <collection name=“VARIATIONS" referenced-type="Allele" reverse-reference="genes" />
>>>
>>> <collection name="genes" referenced-type="Gene" reverse-reference=“VARIATIONS" />
>>>
>>> (caps to show the difference), the build works fine. I am assuming that there is some main class that assigns Allele.gene (and maybe Gene.allele), but I don’t know where. The only mention I see for a Allele.gene is in the genomics.priorities file and in some Flymine code.
>>>
>>>
>>> 2- I made the change above and the build (for these two classes worked OK), but I have not set a mapping XPATH to load to these specific items, but I will re-run to check my debugging log.
>>>
>>> Thanks for the help. I will post some other things about the log when I get it set.
>>>
>>> Cheers
>>> Paulo
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Nov 27, 2018, at 4:12 AM, sergio contrino <[hidden email]> wrote:
>>>>
>>>> dear paul,
>>>> do you have any news on this? any luck with the other debugging options?
>>>> if not we have probably to try to reproduce the issue here.
>>>> thanks
>>>> sergio
>>>>
>>>> On 21/11/2018 18:58, Paulo Nuin wrote:
>>>>> Ciao Sergio
>>>>> The CDS and Transcript are legacy and not used anymore, at least not from the mapping we have for the XML load. We have other collections in Alleles that work perfectly, like Phenotypes and Phnotypes not observed.
>>>>> The mapping files are on
>>>>> https://github.com/WormBase/intermine/tree/im-267/wormmine/support/properties_xpath
>>>>> There might be a change here and there that were not committed but it’s basically these that we are using.
>>>>> Grazie
>>>>> Paulo
>>>>>> On Nov 21, 2018, at 10:44 AM, sergio contrino <[hidden email]> wrote:
>>>>>>
>>>>>> hi paulo,
>>>>>> interesting your generic approach: do your system works for CDSs and trascripts to allele (i.e. it is a problem only for the gene allele relationship)?
>>>>>> could you send a link also to the mapping file(s)?
>>>>>> thanks
>>>>>> sergio
>>>>>>
>>>>>>
>>>>>> On 21/11/2018 14:18, Paulo Nuin wrote:
>>>>>>> Hi Julie
>>>>>>> Here what an Allele item looks like (all *s are NULL)
>>>>>>> -[ RECORD 1 ]---------------+-------------------------------
>>>>>>> othername                   | cewivar00722050
>>>>>>> linkedto                    | *
>>>>>>> interpolatedmapposition     | *
>>>>>>> lastupdated                 | *
>>>>>>> sequencestatus              | Sequenced
>>>>>>> koallele                    | *
>>>>>>> nemaallele                  | *
>>>>>>> natureofvariation           | *
>>>>>>> rearrangement               | *
>>>>>>> status                      | Live
>>>>>>> intermine_method            | *
>>>>>>> nbpallele                   | false
>>>>>>> mutagen                     | *
>>>>>>> amber_uag                   | *
>>>>>>> publicname                  | *
>>>>>>> type                        | Natural_variant
>>>>>>> species                     | Caenorhabditis elegans
>>>>>>> id                          | 1000004
>>>>>>> name                        | *
>>>>>>> productionmethod            | *
>>>>>>> missensechange              | *
>>>>>>> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->
>>>>>>
>>>>>>
>>>>>>> secondaryidentifier         | *
>>>>>>> positiveclone               | *
>>>>>>> naturalvariant              | *
>>>>>>> opal_uga                    | *
>>>>>>> ochre_uaa                   | *
>>>>>>> typeofmutation              | Deletion
>>>>>>> donorsplicesite             | *
>>>>>>> symbol                      | WBVar01994782
>>>>>>> acceptorsplicesite          | *
>>>>>>> primaryidentifier           | WBVar01994782
>>>>>>> detectionmethod             | *
>>>>>>> koconsortiumallele          | false
>>>>>>> organismid                  | *
>>>>>>> sequenceontologytermid      | 1000003
>>>>>>> geneid                      | 1000005
>>>>>>> class                       | org.intermine.model.bio.Allele
>>>>>>> And I have
>>>>>>> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>>>>>>> affectedbyalleles | affectsgenes
>>>>>>> -------------------+--------------
>>>>>>> (0 rows)
>>>>>>> Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at
>>>>>>> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>>>>>>> I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection.
>>>>>>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>>>>>>> [integrate] JDJDDEBUG:: cast type: null
>>>>>>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>>>>>>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>>>>>>> [integrate] JDJDDEBUG:: passed
>>>>>>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>>>>>>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>>>>>>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>>>>>>> [integrate] JDJDDEBUG:: false
>>>>>>> I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.
>>>>>>> Cheers
>>>>>>> Paulo
>>>>>>>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>
>>>>>>>> Paulo,
>>>>>>>>
>>>>>>>> (CCing the dev list for help.)
>>>>>>>>
>>>>>>>> Yes your data model looks correct now!
>>>>>>>>
>>>>>>>> What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?
>>>>>>>>
>>>>>>>> I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:
>>>>>>>>
>>>>>>>> allele.addToCollection("affectsGenes", gene)
>>>>>>>> gene.addToCollection("affectedByAlleles", allele)
>>>>>>>>
>>>>>>>> Here are some docs on many-to-many relationships:
>>>>>>>>
>>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>>>>>>>
>>>>>>>> Does that help?
>>>>>>>> Julie
>>>>>>>>
>>>>>>>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>>>>>>>> Hi Julie
>>>>>>>>> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>>>>>>>>>        <class name="Gene" is-interface="true">
>>>>>>>>>                <attribute name="operon" type="java.lang.String"/>
>>>>>>>>>                <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>>>>>>>>>                <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>>>>>>>>>                <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>>>>>>>                <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>>>>>>>                <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>>>>>>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>>>>>>>>>                <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>>>>>>>>>                <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>>>>>>>>>                <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>>>>>>>        </class>
>>>>>>>>> <class name="Allele" is-interface="true">
>>>>>>>>>          <attribute name="naturalVariant" type="java.lang.String" />
>>>>>>>>>          <attribute name="method" type="java.lang.String" />
>>>>>>>>>          <attribute name="type" type="java.lang.String" />
>>>>>>>>>          <attribute name="publicName" type="java.lang.String" />
>>>>>>>>>          <attribute name="otherName" type="java.lang.String" />
>>>>>>>>>          <attribute name="KOAllele" type="java.lang.String" />
>>>>>>>>>          <attribute name="NBPAllele" type="java.lang.String" />
>>>>>>>>>          <attribute name="nemaAllele" type="java.lang.String" />
>>>>>>>>>          <attribute name="amber_UAG" type="java.lang.String" />
>>>>>>>>>          <attribute name="ochre_UAA" type="java.lang.String" />
>>>>>>>>>          <attribute name="opal_UGA" type="java.lang.String" />
>>>>>>>>>          <attribute name="donorSpliceSite" type="java.lang.String" />
>>>>>>>>>          <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>>>>>>>          <attribute name="productionMethod" type="java.lang.String" />
>>>>>>>>> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>>>>>>>>>          <attribute name="status" type="java.lang.String" />
>>>>>>>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>>>>>>>          <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>>>>>>>>>          <attribute name="positiveClone" type="java.lang.String" />
>>>>>>>>>          <attribute name="typeOfMutation" type="java.lang.String" />
>>>>>>>>>          <attribute name="sequenceStatus" type="java.lang.String" />
>>>>>>>>>          <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>>>>>>>          <attribute name="missenseChange" type="java.lang.String" />
>>>>>>>>>          <attribute name="mutagen" type="java.lang.String" />
>>>>>>>>>          <attribute name="natureOfVariation" type="java.lang.String" />
>>>>>>>>>          <attribute name="interpolatedMapPosition" type="java.lang.String" />
>>>>>>>>>          <attribute name="species" type="java.lang.String" />
>>>>>>>>>          <attribute name="rearrangement" type="java.lang.String" />
>>>>>>>>>          <attribute name="linkedTo" type="java.lang.String" />
>>>>>>>>>          <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>>>>>>>>>          <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
>>>>>>>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>>>>>>>>>          <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>>>>>>>>>          <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>>>>>>>          <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>>>>>>>>>          <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>>>>>>>>>          <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>>>>>>>>>        </class>
>>>>>>>>> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
>>>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>>>>>>>> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
>>>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>>>>>>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>>>>>>>> [integrate] JDJDDEBUG:: xpathvalue:
>>>>>>>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
>>>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>>>> [integrate] JDJDDEBUG:: fieldname=gene
>>>>>>>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>>>>>>>> [integrate] JDJDDEBUG:: N:1
>>>>>>>>> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>>>>>>>>> Thanks again
>>>>>>>>> Paulo
>>>>>>>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Paulo,
>>>>>>>>>>
>>>>>>>>>> For a reverse reference to work, the labels have to match.
>>>>>>>>>>
>>>>>>>>>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>>>>>>>>>
>>>>>>>>>> Does that make sense?
>>>>>>>>>>
>>>>>>>>>> Here are some docs:
>>>>>>>>>>
>>>>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>>>>>>>
>>>>>>>>>> I would correct your data model, run a clean and rebuild and try the query again.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Julie
>>>>>>>>>>
>>>>>>>>>> PS. I'm CCing the dev list because I think this might help other people too!
>>>>>>>>>>
>>>>>>>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>>>>>>>> Hi
>>>>>>>>>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>>>>>>>>>> Thanks
>>>>>>>>>>> Paulo
>>>>>>>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>>>>>>>>>
>>>>>>>>>>>> Is that a typo?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>>>>>>>> Hi Julie
>>>>>>>>>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>>>>>>>>>> Our project addition file is on
>>>>>>>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>>>>>>>> and the Allele class has
>>>>>>>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>>>>>>>> while the Gene has
>>>>>>>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>>>>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>>>>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>>>>>>>> Any help appreciated
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Paulo
>>>>>>> _______________________________________________
>>>>>>> dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>
>>>>>> --
>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>
>>>> --
>>>> sergio contrino                  InterMine, University of Cambridge
>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problems with collections

sergio contrino-2
dear paul,
there are a few places in the code where the relationship is defined as
a 1:N (see below for a couple of examples).
i don't think you are using the chado-db source, but i am not sure about
the acedb-dev genomic model.
are you using those sources? if you do, could you try removing the
conflicting settings to test if this solve your problem?
thanks
sergio

./sources/chado-db/chado-db_additions.xml:    <collection name="alleles"
referenced-type="Allele" reverse-reference="gene"/>
./sources/chado-db/chado-db_additions.xml:    <reference name="gene"
referenced-type="Gene" reverse-reference="alleles"/>

wormmine/support/acedb-dev/im/genomic_model.xml: <collection
name="alleles" referenced-type="Allele" reverse-reference="gene"/>
wormmine/support/acedb-dev/im/genomic_model.xml: <reference name="gene"
referenced-type="Gene" reverse-reference="alleles"/>



On 27/11/2018 18:47, Paulo Nuin wrote:

> Hi
>
> Yes, 267 is the latest, the only difference there would be the genomic additions file that I have changed in the last weeks
>
> Thanks
>
> Paulo
>
>
>
>> On Nov 27, 2018, at 11:44 AM, sergio contrino <[hidden email]> wrote:
>>
>> thank you paul,
>> i think you are right, it seems it is considering this a reference notwithstanding your additions xml file. i'll check your checkout (which i assume is the im-267 branch).
>> thanks
>> sergio
>>
>> On 27/11/2018 16:46, Paulo Nuin wrote:
>>> Hi again
>>> So, I did a quick build of both classes and this is what I see from the log:
>>> Loading Gene
>>> [integrate] JDJDDEBUG:: Retrieving:[primaryIdentifier]
>>> [integrate] JDJDDEBUG:: fieldname=primaryIdentifier
>>> [integrate] JDJDDEBUG:: xpathvalue:WBGene00000898
>>> [integrate] JDJDDEBUG:: Retrieving:[alleles.primaryIdentifier]
>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>> [integrate] JDJDDEBUG:: fieldname=alleles
>>> [integrate] JDJDDEBUG:: This is a collection <-------------------------
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143694
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00143694]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143948
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00143948]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143891
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00143891]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143945
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00143945]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143947
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00143947]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143949
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00143949]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00143967
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00143967]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088550
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00088550]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088710
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00088710]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088714
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00088714]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00088892
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00088892]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242507
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00242507]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242509
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00242509]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242513
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00242513]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242527
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00242527]
>>> [integrate] JDJDDEBUG:: new Allele object:WBVar00242530
>>> [integrate] JDJDDEBUG:: alleles:[WBVar00242530]
>>> It seems to get the Allele items properly.
>>> Loading Allele
>>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>>> [integrate] JDJDDEBUG:: cast type: null
>>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>>> [integrate] JDJDDEBUG:: passed
>>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>>> [integrate] JDJDDEBUG:: false
>>> …..
>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]
>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>> [integrate] JDJDDEBUG:: fieldname=gene
>>> [integrate] JDJDDEBUG:: new Gene object:WBGene00000011
>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000011]
>>> [integrate] JDJDDEBUG:: N:1
>>> It seems that it still doesn’t see it as a collection.
>>> But in the end I don’t see where the relationship ends being created. The Gene and Allele table items are bare and the relations table are empty
>>> intermine_dev_268_small=# select * from genesvariations;
>>>   variations | genes
>>> ------------+-------
>>> (0 rows)
>>> Cheers
>>> Paulo
>>>> On Nov 27, 2018, at 8:37 AM, Paulo Nuin <[hidden email]> wrote:
>>>>
>>>> Ciao Sergio
>>>>
>>>> No, no luck. I have probably run about 100 test builds since we last exchanged emails. Some issues along the way:
>>>>
>>>> 1 - If I enter these in the model
>>>>
>>>>                 <collection name="alleles" referenced-type="Allele" reverse-reference="genes" />
>>>>
>>>> <collection name="genes" referenced-type="Gene" reverse-reference="alleles" />
>>>>
>>>> I get an error on build that says that one reverse reference is missing. If I change to
>>>>
>>>> <collection name=“VARIATIONS" referenced-type="Allele" reverse-reference="genes" />
>>>>
>>>> <collection name="genes" referenced-type="Gene" reverse-reference=“VARIATIONS" />
>>>>
>>>> (caps to show the difference), the build works fine. I am assuming that there is some main class that assigns Allele.gene (and maybe Gene.allele), but I don’t know where. The only mention I see for a Allele.gene is in the genomics.priorities file and in some Flymine code.
>>>>
>>>>
>>>> 2- I made the change above and the build (for these two classes worked OK), but I have not set a mapping XPATH to load to these specific items, but I will re-run to check my debugging log.
>>>>
>>>> Thanks for the help. I will post some other things about the log when I get it set.
>>>>
>>>> Cheers
>>>> Paulo
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Nov 27, 2018, at 4:12 AM, sergio contrino <[hidden email]> wrote:
>>>>>
>>>>> dear paul,
>>>>> do you have any news on this? any luck with the other debugging options?
>>>>> if not we have probably to try to reproduce the issue here.
>>>>> thanks
>>>>> sergio
>>>>>
>>>>> On 21/11/2018 18:58, Paulo Nuin wrote:
>>>>>> Ciao Sergio
>>>>>> The CDS and Transcript are legacy and not used anymore, at least not from the mapping we have for the XML load. We have other collections in Alleles that work perfectly, like Phenotypes and Phnotypes not observed.
>>>>>> The mapping files are on
>>>>>> https://github.com/WormBase/intermine/tree/im-267/wormmine/support/properties_xpath
>>>>>> There might be a change here and there that were not committed but it’s basically these that we are using.
>>>>>> Grazie
>>>>>> Paulo
>>>>>>> On Nov 21, 2018, at 10:44 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>
>>>>>>> hi paulo,
>>>>>>> interesting your generic approach: do your system works for CDSs and trascripts to allele (i.e. it is a problem only for the gene allele relationship)?
>>>>>>> could you send a link also to the mapping file(s)?
>>>>>>> thanks
>>>>>>> sergio
>>>>>>>
>>>>>>>
>>>>>>> On 21/11/2018 14:18, Paulo Nuin wrote:
>>>>>>>> Hi Julie
>>>>>>>> Here what an Allele item looks like (all *s are NULL)
>>>>>>>> -[ RECORD 1 ]---------------+-------------------------------
>>>>>>>> othername                   | cewivar00722050
>>>>>>>> linkedto                    | *
>>>>>>>> interpolatedmapposition     | *
>>>>>>>> lastupdated                 | *
>>>>>>>> sequencestatus              | Sequenced
>>>>>>>> koallele                    | *
>>>>>>>> nemaallele                  | *
>>>>>>>> natureofvariation           | *
>>>>>>>> rearrangement               | *
>>>>>>>> status                      | Live
>>>>>>>> intermine_method            | *
>>>>>>>> nbpallele                   | false
>>>>>>>> mutagen                     | *
>>>>>>>> amber_uag                   | *
>>>>>>>> publicname                  | *
>>>>>>>> type                        | Natural_variant
>>>>>>>> species                     | Caenorhabditis elegans
>>>>>>>> id                          | 1000004
>>>>>>>> name                        | *
>>>>>>>> productionmethod            | *
>>>>>>>> missensechange              | *
>>>>>>>> nemagenetagconsortiumallele | false    <!-- TODO -->    <!-- TODO -->
>>>>>>>
>>>>>>>
>>>>>>>> secondaryidentifier         | *
>>>>>>>> positiveclone               | *
>>>>>>>> naturalvariant              | *
>>>>>>>> opal_uga                    | *
>>>>>>>> ochre_uaa                   | *
>>>>>>>> typeofmutation              | Deletion
>>>>>>>> donorsplicesite             | *
>>>>>>>> symbol                      | WBVar01994782
>>>>>>>> acceptorsplicesite          | *
>>>>>>>> primaryidentifier           | WBVar01994782
>>>>>>>> detectionmethod             | *
>>>>>>>> koconsortiumallele          | false
>>>>>>>> organismid                  | *
>>>>>>>> sequenceontologytermid      | 1000003
>>>>>>>> geneid                      | 1000005
>>>>>>>> class                       | org.intermine.model.bio.Allele
>>>>>>>> And I have
>>>>>>>> intermine_dev_268_small=# select * from affectedbyallelesaffectsgenes;
>>>>>>>> affectedbyalleles | affectsgenes
>>>>>>>> -------------------+--------------
>>>>>>>> (0 rows)
>>>>>>>> Our parser is generic for 10 or more Xml files and most of other collections are being set properly. We have 38 testing queries, most of them testing collections and this is the only that fails. Parser is at
>>>>>>>> https://github.com/WormBase/intermine/blob/im-266/bio/sources/wormbase-acedb/main/src/org/intermine/bio/dataconversion/WormbaseAcedbConverter.java
>>>>>>>> I see this from the loader log. Gene is being cast as NULL. I think I need to create a collection Gene instead of having the affectedByAllele collection.
>>>>>>>> [integrate] JDJDDEBUG:: === gene.primaryIdentifier ===
>>>>>>>> [integrate] JDJDDEBUG:: cast type: null
>>>>>>>> [integrate] JDJDDEBUG:: datapath: gene.primaryIdentifier
>>>>>>>> [integrate] JDJDDEBUG:: gene.primaryIdentifier
>>>>>>>> [integrate] JDJDDEBUG:: passed
>>>>>>>> [integrate] JDJDDEBUG:: /Variation/Affects/Gene/Gene/text()
>>>>>>>> [integrate] JDJDDEBUG:: new1 primaryIdentifier
>>>>>>>> [integrate] JDJDDEBUG:: new2 primaryIdentifier
>>>>>>>> [integrate] JDJDDEBUG:: false
>>>>>>>> I have some other debugging options, will try a new collection and uncomment the debug statements and get back to you.
>>>>>>>> Cheers
>>>>>>>> Paulo
>>>>>>>>> On Nov 21, 2018, at 1:17 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>
>>>>>>>>> Paulo,
>>>>>>>>>
>>>>>>>>> (CCing the dev list for help.)
>>>>>>>>>
>>>>>>>>> Yes your data model looks correct now!
>>>>>>>>>
>>>>>>>>> What does the data look like after your build? Are both collections (affectedByAlleles, affectsGenes) empty?
>>>>>>>>>
>>>>>>>>> I would check your data parser. Are you setting the collections correctly? For example, you should have one of these statements:
>>>>>>>>>
>>>>>>>>> allele.addToCollection("affectsGenes", gene)
>>>>>>>>> gene.addToCollection("affectedByAlleles", allele)
>>>>>>>>>
>>>>>>>>> Here are some docs on many-to-many relationships:
>>>>>>>>>
>>>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/#types-of-relationship
>>>>>>>>>
>>>>>>>>> Does that help?
>>>>>>>>> Julie
>>>>>>>>>
>>>>>>>>> On 20/11/2018 13:48, Paulo Nuin wrote:
>>>>>>>>>> Hi Julie
>>>>>>>>>> Still have\ing errors with this. I made some changes to my Gene and Allele classes and affectedByAlleles is linked as a collection to affectsGenes. There are no plural or singular differences
>>>>>>>>>>         <class name="Gene" is-interface="true">
>>>>>>>>>>                 <attribute name="operon" type="java.lang.String"/>
>>>>>>>>>>                 <!--<reference name="referenceAllele" referenced-type="Allele" />-->
>>>>>>>>>>                 <collection name="strains" referenced-type="Strain" reverse-reference="genes"/>
>>>>>>>>>>                 <collection name="expressionPatterns" referenced-type="ExpressionPattern" reverse-reference="gene"/>
>>>>>>>>>>                 <collection name="expressionClusters" referenced-type="ExpressionCluster" reverse-reference="genes"/>
>>>>>>>>>>                 <collection name="regulatesExprCluster" referenced-type="ExpressionCluster" reverse-reference="regulatedByGene"/>
>>>>>>>>>> —>>>>>  <collection name="affectedByAlleles" referenced-type="Allele" reverse-reference="affectsGenes" />
>>>>>>>>>>                 <collection name="RNAiResult" referenced-type="RNAi" reverse-reference="inhibitsGene"/>
>>>>>>>>>>                 <collection name="geneClass"  referenced-type="GeneClass" reverse-reference="genes"/>
>>>>>>>>>>                 <collection name="formerGeneClass"  referenced-type="GeneClass" reverse-reference="formerGenes"/>
>>>>>>>>>>         </class>
>>>>>>>>>> <class name="Allele" is-interface="true">
>>>>>>>>>>           <attribute name="naturalVariant" type="java.lang.String" />
>>>>>>>>>>           <attribute name="method" type="java.lang.String" />
>>>>>>>>>>           <attribute name="type" type="java.lang.String" />
>>>>>>>>>>           <attribute name="publicName" type="java.lang.String" />
>>>>>>>>>>           <attribute name="otherName" type="java.lang.String" />
>>>>>>>>>>           <attribute name="KOAllele" type="java.lang.String" />
>>>>>>>>>>           <attribute name="NBPAllele" type="java.lang.String" />
>>>>>>>>>>           <attribute name="nemaAllele" type="java.lang.String" />
>>>>>>>>>>           <attribute name="amber_UAG" type="java.lang.String" />
>>>>>>>>>>           <attribute name="ochre_UAA" type="java.lang.String" />
>>>>>>>>>>           <attribute name="opal_UGA" type="java.lang.String" />
>>>>>>>>>>           <attribute name="donorSpliceSite" type="java.lang.String" />
>>>>>>>>>>           <attribute name="acceptorSpliceSite" type="java.lang.String" />
>>>>>>>>>>           <attribute name="productionMethod" type="java.lang.String" />
>>>>>>>>>> <!--      <attribute name="sequenceDetails" type="java.lang.String" /> -->
>>>>>>>>>>           <attribute name="status" type="java.lang.String" />
>>>>>>>>>> <!--      <attribute name="affects" type="java.lang.String" /> -->
>>>>>>>>>>           <attribute name="NemaGENETAGConsortiumAllele" type="java.lang.String" />          <attribute name="detectionMethod" type="java.lang.String" />
>>>>>>>>>>           <attribute name="positiveClone" type="java.lang.String" />
>>>>>>>>>>           <attribute name="typeOfMutation" type="java.lang.String" />
>>>>>>>>>>           <attribute name="sequenceStatus" type="java.lang.String" />
>>>>>>>>>>           <attribute name="KOConsortiumAllele" type="java.lang.String" />
>>>>>>>>>>           <attribute name="missenseChange" type="java.lang.String" />
>>>>>>>>>>           <attribute name="mutagen" type="java.lang.String" />
>>>>>>>>>>           <attribute name="natureOfVariation" type="java.lang.String" />
>>>>>>>>>>           <attribute name="interpolatedMapPosition" type="java.lang.String" />
>>>>>>>>>>           <attribute name="species" type="java.lang.String" />
>>>>>>>>>>           <attribute name="rearrangement" type="java.lang.String" />
>>>>>>>>>>           <attribute name="linkedTo" type="java.lang.String" />
>>>>>>>>>>           <collection name="phenotype" referenced-type="Phenotype" reverse-reference="phenotypeAllele" />
>>>>>>>>>>           <collection name="phenotypeNotObserved" referenced-type="Phenotype" reverse-reference="phenotypeAlleleNotObserved" />
>>>>>>>>>> —>>>>>>  <collection name="affectsGenes" referenced-type="Gene" reverse-reference="affectedByAlleles" />
>>>>>>>>>>           <!--<collection name="affectsCDSs" referenced-type="CDS" reverse-reference="affectedByAlleles" />
>>>>>>>>>>           <collection name="affectsTranscripts" referenced-type="Transcript" reverse-reference="affectedByAlleles" />
>>>>>>>>>>           <collection name="strains" referenced-type="Strain" reverse-reference="variations" /> -->
>>>>>>>>>>           <collection name="fromLabs" referenced-type="Laboratory" reverse-reference="variations" />
>>>>>>>>>>           <collection name="geneClass" referenced-type="GeneClass" reverse-reference="variations" />
>>>>>>>>>>         </class>
>>>>>>>>>> As I mentioned before I have the same identical structure for phenotypes, but when I run the build the logs are really different for each one:
>>>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[phenotypeNotObserved.primaryIdentifier]    <<<<<<<------------
>>>>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>>>>> [integrate] JDJDDEBUG:: fieldname=phenotypeNotObserved
>>>>>>>>>> [integrate] JDJDDEBUG:: This is a collection      <<<<<<<<<<<<<<,-----------------
>>>>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[opal_UGA]
>>>>>>>>>> [integrate] JDJDDEBUG:: fieldname=opal_UGA
>>>>>>>>>> [integrate] JDJDDEBUG:: xpathvalue:
>>>>>>>>>> [integrate] JDJDDEBUG:: ignoring attribute [opal_UGA], no value
>>>>>>>>>> [integrate] JDJDDEBUG:: =======================
>>>>>>>>>> [integrate] JDJDDEBUG:: Retrieving:[gene.primaryIdentifier]   <<<<<<<<<<<<<<<-----------------
>>>>>>>>>> [integrate] JDJDDEBUG:: suffix:primaryIdentifier
>>>>>>>>>> [integrate] JDJDDEBUG:: fieldname=gene
>>>>>>>>>> [integrate] JDJDDEBUG:: Setting current Allele.gene to: (Gene)[WBGene00000898]    <<<<<<<<<<<<-------------
>>>>>>>>>> [integrate] JDJDDEBUG:: N:1
>>>>>>>>>> Am I missing any other place that might be setting this? Why the Allele.gene is being created and it’s not a collection?
>>>>>>>>>> Thanks again
>>>>>>>>>> Paulo
>>>>>>>>>>> On Nov 13, 2018, at 1:15 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Paulo,
>>>>>>>>>>>
>>>>>>>>>>> For a reverse reference to work, the labels have to match.
>>>>>>>>>>>
>>>>>>>>>>> If you have "genes" as the collection, then "genes" (not "gene"!) has to be the value of the reverse reference to that collection.
>>>>>>>>>>>
>>>>>>>>>>> Does that make sense?
>>>>>>>>>>>
>>>>>>>>>>> Here are some docs:
>>>>>>>>>>>
>>>>>>>>>>> https://intermine.readthedocs.io/en/latest/data-model/model/
>>>>>>>>>>>
>>>>>>>>>>> I would correct your data model, run a clean and rebuild and try the query again.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Julie
>>>>>>>>>>>
>>>>>>>>>>> PS. I'm CCing the dev list because I think this might help other people too!
>>>>>>>>>>>
>>>>>>>>>>> On 12/11/2018 16:43, Paulo Nuin wrote:
>>>>>>>>>>>> Hi
>>>>>>>>>>>> I am pretty sure I have tried with gene and genes, and it didn’t work. Anything else I would need to check?
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Paulo
>>>>>>>>>>>>> On Nov 12, 2018, at 8:42 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Those reverse references don't match. You have a "gene" collection but the reverse reference is "genes".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is that a typo?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 12/11/2018 15:37, Paulo Nuin wrote:
>>>>>>>>>>>>>> Hi Julie
>>>>>>>>>>>>>> We are having some problems with many-to-many relationships between a couple of sources. The Allele and Gene classes are supposed to have many items with relationships, which should be seen on query results, but we are only obtaining one gene per allele (variation).
>>>>>>>>>>>>>> Our project addition file is on
>>>>>>>>>>>>>> https://github.com/WormBase/intermine/blob/im-267/bio/sources/wormbase-acedb/wormbase-acedb_additions.xml
>>>>>>>>>>>>>> and the Allele class has
>>>>>>>>>>>>>> <collectionname="gene"referenced-type="Gene"reverse-reference="variations"/>
>>>>>>>>>>>>>> while the Gene has
>>>>>>>>>>>>>> <collectionname="variations"referenced-type="Allele"reverse-reference="genes"/>
>>>>>>>>>>>>>> would this be correct? I have tried all possibilities in testing, and worked a lot on the mapping and how these classes and elements are loaded in the DB, with much success. A representative query would be
>>>>>>>>>>>>>> <query name="" model="genomic" view="Allele.primaryIdentifier Allele.symbol Allele.gene.primaryIdentifier Allele.gene.secondaryIdentifier Allele.gene.symbol" longDescription="" sortOrder="Allele.primaryIdentifier asc"> <constraint path="Allele.symbol" op="=" value="gk962622"/> </query>
>>>>>>>>>>>>>> but gk962622 just shows one gene while it should be linked to many.
>>>>>>>>>>>>>> Any help appreciated
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Paulo
>>>>>>>> _______________________________________________
>>>>>>>> dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>
>>>>>>> --
>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>
>>>>> --
>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev