Starting on InterMine 2.0....
Locked
 
|
Hello,
I'm (finally) trying to build InterMine 2.0. But a couple of things have popped up when I tried your testmine demo: 1) The testmine demo does not appear to use anything other than localhost for the db server. There are a couple edits needed in setup.sh and testmodel.properties: > testmine> git diff dbmodel/resources/testmodel.properties > diff --git a/testmine/dbmodel/resources/testmodel.properties > b/testmine/dbmodel/resources/testmodel.properties > index e3ae168..488bcd2 100644 > --- a/testmine/dbmodel/resources/testmodel.properties > +++ b/testmine/dbmodel/resources/testmodel.properties > @@ -1,9 +1,9 @@ > -db.userprofile-test.datasource.serverName=localhost > +db.userprofile-test.datasource.serverName=SERVER > db.userprofile-test.datasource.databaseName=userprofile-demo > db.userprofile-test.datasource.user=PSQL_USER > db.userprofile-test.datasource.password=PSQL_PWD > > -db.unittest.datasource.serverName=localhost > +db.unittest.datasource.serverName=SERVER > db.unittest.datasource.databaseName=intermine-demo > db.unittest.datasource.user=PSQL_USER > db.unittest.datasource.password=PSQL_PWD > testmine> git diff setup.sh > diff --git a/testmine/setup.sh b/testmine/setup.sh > index 0d13f74..41bee60 100755 > --- a/testmine/setup.sh > +++ b/testmine/setup.sh > @@ -74,11 +74,11 @@ fi > > echo "------> Checking databases..." > for db in $USERPROFILEDB $PRODDB; do > - if psql --list | egrep -q '\s'$db'\s'; then > + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then > echo $db exists. > else > echo Creating $db > - createdb $db > + createdb -h $SERVER -E UTF8 $db > fi > done (you want a pull request?) 2) The other thing is that the build tripping over UTF-8 characters. There are a couple in the source comments (those are trivial), but the more important one was that I got an 'invalid byte sequence error' when inserting names; I assume it's Robert?? Setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. Thanks, Joe _______________________________________________ dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
1. Whoops, yes, thanks for that. I'll fix. We only use testmine for
travis so I didn't catch that -- but that makes sense to use testmine on not-localhost. 2. Thanks for the fix! Not sure why you are seeing this error and we aren't though. On 20/10/18 00:54, Joe Carlson wrote: > Hello, > > I'm (finally) trying to build InterMine 2.0. But a couple of things have > popped up when I tried your testmine demo: > > 1) The testmine demo does not appear to use anything other than > localhost for the db server. There are a couple edits needed in setup.sh > and testmodel.properties: > >> testmine> git diff dbmodel/resources/testmodel.properties >> diff --git a/testmine/dbmodel/resources/testmodel.properties >> b/testmine/dbmodel/resources/testmodel.properties >> index e3ae168..488bcd2 100644 >> --- a/testmine/dbmodel/resources/testmodel.properties >> +++ b/testmine/dbmodel/resources/testmodel.properties >> @@ -1,9 +1,9 @@ >> -db.userprofile-test.datasource.serverName=localhost >> +db.userprofile-test.datasource.serverName=SERVER >> db.userprofile-test.datasource.databaseName=userprofile-demo >> db.userprofile-test.datasource.user=PSQL_USER >> db.userprofile-test.datasource.password=PSQL_PWD >> >> -db.unittest.datasource.serverName=localhost >> +db.unittest.datasource.serverName=SERVER >> db.unittest.datasource.databaseName=intermine-demo >> db.unittest.datasource.user=PSQL_USER >> db.unittest.datasource.password=PSQL_PWD >> testmine> git diff setup.sh >> diff --git a/testmine/setup.sh b/testmine/setup.sh >> index 0d13f74..41bee60 100755 >> --- a/testmine/setup.sh >> +++ b/testmine/setup.sh >> @@ -74,11 +74,11 @@ fi >> >> echo "------> Checking databases..." >> for db in $USERPROFILEDB $PRODDB; do >> - if psql --list | egrep -q '\s'$db'\s'; then >> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >> echo $db exists. >> else >> echo Creating $db >> - createdb $db >> + createdb -h $SERVER -E UTF8 $db >> fi >> done > > (you want a pull request?) > > 2) The other thing is that the build tripping over UTF-8 characters. > There are a couple in the source comments (those are trivial), but the > more important one was that I got an 'invalid byte sequence error' when > inserting names; I assume it's Robert?? Setting the environment variable > JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. > > Thanks, > > Joe > > _______________________________________________ > dev mailing list > [hidden email] > https://lists.intermine.org/mailman/listinfo/dev dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
What I read from the docs is that the “platform default” character encoding is used. What do you see for the output of ‘locale’? I have LANG="en_US.UTF-8” but also 'LC_CTYPE=“C”. I don’t know which setting javac picks up; but this might be the issue. Joe
_______________________________________________ dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
In reply to this post by Julie Sullivan-2
Hi Julie,
I'm back looking at the gradle build. A couple other things popped up: 1) setup.sh in biotestmine is also not coded for a non-localhost postgres server. 2) the more puzzling thing (for me, at least) is that gradle.properties specifies intermine plugin version 3.0+. From what I see, these have not been uploaded to jcenter (2.1.1 seems to be the latest). Or am I doing something wrong here? Joe On 10/23/2018 02:26 AM, Julie Sullivan wrote: > 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for > travis so I didn't catch that -- but that makes sense to use testmine > on not-localhost. > > 2. Thanks for the fix! Not sure why you are seeing this error and we > aren't though. > > On 20/10/18 00:54, Joe Carlson wrote: >> Hello, >> >> I'm (finally) trying to build InterMine 2.0. But a couple of things >> have popped up when I tried your testmine demo: >> >> 1) The testmine demo does not appear to use anything other than >> localhost for the db server. There are a couple edits needed in >> setup.sh and testmodel.properties: >> >>> testmine> git diff dbmodel/resources/testmodel.properties >>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>> b/testmine/dbmodel/resources/testmodel.properties >>> index e3ae168..488bcd2 100644 >>> --- a/testmine/dbmodel/resources/testmodel.properties >>> +++ b/testmine/dbmodel/resources/testmodel.properties >>> @@ -1,9 +1,9 @@ >>> -db.userprofile-test.datasource.serverName=localhost >>> +db.userprofile-test.datasource.serverName=SERVER >>> db.userprofile-test.datasource.databaseName=userprofile-demo >>> db.userprofile-test.datasource.user=PSQL_USER >>> db.userprofile-test.datasource.password=PSQL_PWD >>> >>> -db.unittest.datasource.serverName=localhost >>> +db.unittest.datasource.serverName=SERVER >>> db.unittest.datasource.databaseName=intermine-demo >>> db.unittest.datasource.user=PSQL_USER >>> db.unittest.datasource.password=PSQL_PWD >>> testmine> git diff setup.sh >>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>> index 0d13f74..41bee60 100755 >>> --- a/testmine/setup.sh >>> +++ b/testmine/setup.sh >>> @@ -74,11 +74,11 @@ fi >>> >>> echo "------> Checking databases..." >>> for db in $USERPROFILEDB $PRODDB; do >>> - if psql --list | egrep -q '\s'$db'\s'; then >>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>> echo $db exists. >>> else >>> echo Creating $db >>> - createdb $db >>> + createdb -h $SERVER -E UTF8 $db >>> fi >>> done >> >> (you want a pull request?) >> >> 2) The other thing is that the build tripping over UTF-8 characters. >> There are a couple in the source comments (those are trivial), but >> the more important one was that I got an 'invalid byte sequence >> error' when inserting names; I assume it's Robert?? Setting the >> environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears >> to have fixed that. >> >> Thanks, >> >> Joe >> >> _______________________________________________ >> dev mailing list >> [hidden email] >> https://lists.intermine.org/mailman/listinfo/dev _______________________________________________ dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
Hi Joe!
1. Yes, this is on purpose (for Travis) but you can have a local biotestmine.properties that will be used instead of the default: https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 So the default properties file has localhost, but you can use your own properties file if you need different settings. Is that sufficient? Again, this script was written for travis but I can update. 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got overzealous. You are right, 2.1.1 is the latest release. You are not doing anything wrong! :) Julie On 02/11/2018 22:25, Joe Carlson wrote: > Hi Julie, > > I'm back looking at the gradle build. A couple other things popped up: > > 1) setup.sh in biotestmine is also not coded for a non-localhost > postgres server. > > 2) the more puzzling thing (for me, at least) is that gradle.properties > specifies intermine plugin version 3.0+. From what I see, these have not > been uploaded to jcenter (2.1.1 seems to be the latest). > > Or am I doing something wrong here? > > Joe > > On 10/23/2018 02:26 AM, Julie Sullivan wrote: >> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for >> travis so I didn't catch that -- but that makes sense to use testmine >> on not-localhost. >> >> 2. Thanks for the fix! Not sure why you are seeing this error and we >> aren't though. >> >> On 20/10/18 00:54, Joe Carlson wrote: >>> Hello, >>> >>> I'm (finally) trying to build InterMine 2.0. But a couple of things >>> have popped up when I tried your testmine demo: >>> >>> 1) The testmine demo does not appear to use anything other than >>> localhost for the db server. There are a couple edits needed in >>> setup.sh and testmodel.properties: >>> >>>> testmine> git diff dbmodel/resources/testmodel.properties >>>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>>> b/testmine/dbmodel/resources/testmodel.properties >>>> index e3ae168..488bcd2 100644 >>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>> @@ -1,9 +1,9 @@ >>>> -db.userprofile-test.datasource.serverName=localhost >>>> +db.userprofile-test.datasource.serverName=SERVER >>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>> db.userprofile-test.datasource.user=PSQL_USER >>>> db.userprofile-test.datasource.password=PSQL_PWD >>>> >>>> -db.unittest.datasource.serverName=localhost >>>> +db.unittest.datasource.serverName=SERVER >>>> db.unittest.datasource.databaseName=intermine-demo >>>> db.unittest.datasource.user=PSQL_USER >>>> db.unittest.datasource.password=PSQL_PWD >>>> testmine> git diff setup.sh >>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>> index 0d13f74..41bee60 100755 >>>> --- a/testmine/setup.sh >>>> +++ b/testmine/setup.sh >>>> @@ -74,11 +74,11 @@ fi >>>> >>>> echo "------> Checking databases..." >>>> for db in $USERPROFILEDB $PRODDB; do >>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>> echo $db exists. >>>> else >>>> echo Creating $db >>>> - createdb $db >>>> + createdb -h $SERVER -E UTF8 $db >>>> fi >>>> done >>> >>> (you want a pull request?) >>> >>> 2) The other thing is that the build tripping over UTF-8 characters. >>> There are a couple in the source comments (those are trivial), but >>> the more important one was that I got an 'invalid byte sequence >>> error' when inserting names; I assume it's Robert?? Setting the >>> environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears >>> to have fixed that. >>> >>> Thanks, >>> >>> Joe >>> >>> _______________________________________________ >>> dev mailing list >>> [hidden email] >>> https://lists.intermine.org/mailman/listinfo/dev > dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
On 11/05/2018 12:17 AM, Julie Sullivan wrote: > Hi Joe! > > 1. Yes, this is on purpose (for Travis) but you can have a local > biotestmine.properties that will be used instead of the default: > > https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 > > So the default properties file has localhost, but you can use your own > properties file if you need different settings. Is that sufficient? > Again, this script was written for travis but I can update. fail. I don't know if you intend this script to be run by mortals or not. setup.sh has commands for checking and creating the database if needed (lines 94 and 98). These commands to not specify a postgresql host. Yes, you could create the db's manually ahead of time, but the script will fail if you do not have a server running locally. (adding a -h $SERVER will fix it.) Joe > > 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got > overzealous. You are right, 2.1.1 is the latest release. > > You are not doing anything wrong! :) I'm still new to gradle. Just making sure I'm getting it. > Julie > > On 02/11/2018 22:25, Joe Carlson wrote: >> Hi Julie, >> >> I'm back looking at the gradle build. A couple other things popped up: >> >> 1) setup.sh in biotestmine is also not coded for a non-localhost >> postgres server. >> >> 2) the more puzzling thing (for me, at least) is that >> gradle.properties specifies intermine plugin version 3.0+. From what >> I see, these have not been uploaded to jcenter (2.1.1 seems to be the >> latest). >> >> Or am I doing something wrong here? >> >> Joe >> >> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for >>> travis so I didn't catch that -- but that makes sense to use >>> testmine on not-localhost. >>> >>> 2. Thanks for the fix! Not sure why you are seeing this error and we >>> aren't though. >>> >>> On 20/10/18 00:54, Joe Carlson wrote: >>>> Hello, >>>> >>>> I'm (finally) trying to build InterMine 2.0. But a couple of things >>>> have popped up when I tried your testmine demo: >>>> >>>> 1) The testmine demo does not appear to use anything other than >>>> localhost for the db server. There are a couple edits needed in >>>> setup.sh and testmodel.properties: >>>> >>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>>>> b/testmine/dbmodel/resources/testmodel.properties >>>>> index e3ae168..488bcd2 100644 >>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>> @@ -1,9 +1,9 @@ >>>>> -db.userprofile-test.datasource.serverName=localhost >>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>> >>>>> -db.unittest.datasource.serverName=localhost >>>>> +db.unittest.datasource.serverName=SERVER >>>>> db.unittest.datasource.databaseName=intermine-demo >>>>> db.unittest.datasource.user=PSQL_USER >>>>> db.unittest.datasource.password=PSQL_PWD >>>>> testmine> git diff setup.sh >>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>> index 0d13f74..41bee60 100755 >>>>> --- a/testmine/setup.sh >>>>> +++ b/testmine/setup.sh >>>>> @@ -74,11 +74,11 @@ fi >>>>> >>>>> echo "------> Checking databases..." >>>>> for db in $USERPROFILEDB $PRODDB; do >>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>> echo $db exists. >>>>> else >>>>> echo Creating $db >>>>> - createdb $db >>>>> + createdb -h $SERVER -E UTF8 $db >>>>> fi >>>>> done >>>> >>>> (you want a pull request?) >>>> >>>> 2) The other thing is that the build tripping over UTF-8 >>>> characters. There are a couple in the source comments (those are >>>> trivial), but the more important one was that I got an 'invalid >>>> byte sequence error' when inserting names; I assume it's Robert?? >>>> Setting the environment variable >>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. >>>> >>>> Thanks, >>>> >>>> Joe >>>> >>>> _______________________________________________ >>>> dev mailing list >>>> [hidden email] >>>> https://lists.intermine.org/mailman/listinfo/dev >> _______________________________________________ dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
In reply to this post by Julie Sullivan-2
Hi Julie, I'm still migrating to gradle. I have a question about the global
additions to the data model. You have this in the docs: Global Additions FilePreviously the data model was merged from all data sources’ additions XML file (plus the SO terms, core and genomic additons). This is no longer true. Since each source is in its own JAR now, the data model is self-contained for each data source. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source: // [in bio/sources/build.gradle]
// uncomment to specify an extra additions file for your bio-sources
// this file will be merged with the additions file for each data source
// and included in each source JAR.
//bioSourceDBModelConfig {
// extraAdditionsFile = "MY-MINE_additions.xml"
//}
which build.gradle file are you
referring to? I have a directory phytomine/ from the make_mine
script; I assumed it was in this build.gradle file (there is not a
bio/sources/ directory below this.), but trying to build the db
was giving the message "Could not find method
bioSourceDBModelConfig()". Also tried to put it in the top level
of my phytomine-bio-sources. Same story.
Thanks,
Joe
On 11/5/18 12:17 AM, Julie Sullivan
wrote:
Hi Joe! _______________________________________________ dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
Hi Joe
The top level of your phytomine's bio/sources directory is the correct placement for that additions file, and that parameter! However the docs were out of date, I renamed that parameter to be: globalAdditionsFile because I thought it would be less confusing! :( I've now updated the docs, apologies. You can look at flymine here: https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 globalAdditionsFile = "flymine-global_additions.xml" And that file (flymine-global_additions.xml) lives in the root directory of flymine-bio-sources. Let me know if you have any more issues! Julie On 13/11/2018 01:49, Joe Carlson wrote: > Hi Julie, > > I'm still migrating to gradle. I have a question about the global > additions to the data model. > > You have this in the docs: > > > Global Additions File > > Previously the data model was merged from all data sources’ additions > XML file (plus the SO terms, core and genomic additons). This is no > longer true. Since each source is in its own JAR now, the data model is > self-contained for each data source. Therefore if you reference a class > in your data parser, it must be present in the additions file. > Alternatively, you can specify a single data model file that will be > merged into each source: > > //[in bio/sources/build.gradle] > // uncomment to specify an extra additions filefor your bio-sources > // this file will be merged with the additions filefor each datasource > // and included in eachsource JAR. > //bioSourceDBModelConfig{ > //extraAdditionsFile = "MY-MINE_additions.xml" > //} > > > > which build.gradle file are you referring to? I have a directory > phytomine/ from the make_mine script; I assumed it was in this > build.gradle file (there is not a bio/sources/ directory below this.), > but trying to build the db was giving the message "Could not find method > bioSourceDBModelConfig()". Also tried to put it in the top level of my > phytomine-bio-sources. Same story. > > Thanks, > > Joe > > On 11/5/18 12:17 AM, Julie Sullivan wrote: >> Hi Joe! >> >> 1. Yes, this is on purpose (for Travis) but you can have a local >> biotestmine.properties that will be used instead of the default: >> >> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 >> >> So the default properties file has localhost, but you can use your own >> properties file if you need different settings. Is that sufficient? >> Again, this script was written for travis but I can update. >> >> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got >> overzealous. You are right, 2.1.1 is the latest release. >> >> You are not doing anything wrong! :) >> Julie >> >> On 02/11/2018 22:25, Joe Carlson wrote: >>> Hi Julie, >>> >>> I'm back looking at the gradle build. A couple other things popped up: >>> >>> 1) setup.sh in biotestmine is also not coded for a non-localhost >>> postgres server. >>> >>> 2) the more puzzling thing (for me, at least) is that >>> gradle.properties specifies intermine plugin version 3.0+. From what >>> I see, these have not been uploaded to jcenter (2.1.1 seems to be the >>> latest). >>> >>> Or am I doing something wrong here? >>> >>> Joe >>> >>> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for >>>> travis so I didn't catch that -- but that makes sense to use >>>> testmine on not-localhost. >>>> >>>> 2. Thanks for the fix! Not sure why you are seeing this error and we >>>> aren't though. >>>> >>>> On 20/10/18 00:54, Joe Carlson wrote: >>>>> Hello, >>>>> >>>>> I'm (finally) trying to build InterMine 2.0. But a couple of things >>>>> have popped up when I tried your testmine demo: >>>>> >>>>> 1) The testmine demo does not appear to use anything other than >>>>> localhost for the db server. There are a couple edits needed in >>>>> setup.sh and testmodel.properties: >>>>> >>>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>>>>> b/testmine/dbmodel/resources/testmodel.properties >>>>>> index e3ae168..488bcd2 100644 >>>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>>> @@ -1,9 +1,9 @@ >>>>>> -db.userprofile-test.datasource.serverName=localhost >>>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>>> >>>>>> -db.unittest.datasource.serverName=localhost >>>>>> +db.unittest.datasource.serverName=SERVER >>>>>> db.unittest.datasource.databaseName=intermine-demo >>>>>> db.unittest.datasource.user=PSQL_USER >>>>>> db.unittest.datasource.password=PSQL_PWD >>>>>> testmine> git diff setup.sh >>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>>> index 0d13f74..41bee60 100755 >>>>>> --- a/testmine/setup.sh >>>>>> +++ b/testmine/setup.sh >>>>>> @@ -74,11 +74,11 @@ fi >>>>>> >>>>>> echo "------> Checking databases..." >>>>>> for db in $USERPROFILEDB $PRODDB; do >>>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>>> echo $db exists. >>>>>> else >>>>>> echo Creating $db >>>>>> - createdb $db >>>>>> + createdb -h $SERVER -E UTF8 $db >>>>>> fi >>>>>> done >>>>> >>>>> (you want a pull request?) >>>>> >>>>> 2) The other thing is that the build tripping over UTF-8 >>>>> characters. There are a couple in the source comments (those are >>>>> trivial), but the more important one was that I got an 'invalid >>>>> byte sequence error' when inserting names; I assume it's Robert?? >>>>> Setting the environment variable >>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. >>>>> >>>>> Thanks, >>>>> >>>>> Joe >>>>> >>>>> _______________________________________________ >>>>> dev mailing list >>>>> [hidden email] >>>>> https://lists.intermine.org/mailman/listinfo/dev >>> dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
Hi Julie,
I think I see what my issues were. I was a bit confused by the docs. One issue was that the https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script say make_source is invoked as ./make_source. But this script requires the skeletons directory below it and as far as I can tell needs to be invokes as /path/to/intermine-scripts/make_source. The second issue was that https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file refers to bio/sources/build.gradle. I was interpreting this as a path and kept looking for the sources folder inside bio. Perhaps this should be labeled as bio-sources or MYMINE-bio-sources, or something. And the final thing is that it seems that bioSourceDBModelConfig specific needs to be within the subprojects section of build.gradle. It I put it at top level, I get the 'Could not find method bioSourceDBModelConfig() for arguments...' error. Thanks, Joe On 11/13/18 1:06 AM, Julie Sullivan wrote: > Hi Joe > > The top level of your phytomine's bio/sources directory is the correct > placement for that additions file, and that parameter! However the > docs were out of date, I renamed that parameter to be: > > globalAdditionsFile > > because I thought it would be less confusing! :( I've now updated the > docs, apologies. > > You can look at flymine here: > > https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 > > > globalAdditionsFile = "flymine-global_additions.xml" > > And that file (flymine-global_additions.xml) lives in the root > directory of flymine-bio-sources. > > Let me know if you have any more issues! > Julie > > > On 13/11/2018 01:49, Joe Carlson wrote: >> Hi Julie, >> >> I'm still migrating to gradle. I have a question about the global >> additions to the data model. >> >> You have this in the docs: >> >> >> Global Additions File >> >> Previously the data model was merged from all data sources’ additions >> XML file (plus the SO terms, core and genomic additons). This is no >> longer true. Since each source is in its own JAR now, the data model >> is self-contained for each data source. Therefore if you reference a >> class in your data parser, it must be present in the additions file. >> Alternatively, you can specify a single data model file that will be >> merged into each source: >> >> //[in bio/sources/build.gradle] >> // uncomment to specify an extra additions filefor your bio-sources >> // this file will be merged with the additions filefor each datasource >> // and included in eachsource JAR. >> //bioSourceDBModelConfig{ >> //extraAdditionsFile = "MY-MINE_additions.xml" >> //} >> >> >> >> which build.gradle file are you referring to? I have a directory >> phytomine/ from the make_mine script; I assumed it was in this >> build.gradle file (there is not a bio/sources/ directory below >> this.), but trying to build the db was giving the message "Could not >> find method bioSourceDBModelConfig()". Also tried to put it in the >> top level of my phytomine-bio-sources. Same story. >> >> Thanks, >> >> Joe >> >> On 11/5/18 12:17 AM, Julie Sullivan wrote: >>> Hi Joe! >>> >>> 1. Yes, this is on purpose (for Travis) but you can have a local >>> biotestmine.properties that will be used instead of the default: >>> >>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 >>> >>> So the default properties file has localhost, but you can use your >>> own properties file if you need different settings. Is that >>> sufficient? Again, this script was written for travis but I can update. >>> >>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got >>> overzealous. You are right, 2.1.1 is the latest release. >>> >>> You are not doing anything wrong! :) >>> Julie >>> >>> On 02/11/2018 22:25, Joe Carlson wrote: >>>> Hi Julie, >>>> >>>> I'm back looking at the gradle build. A couple other things popped up: >>>> >>>> 1) setup.sh in biotestmine is also not coded for a non-localhost >>>> postgres server. >>>> >>>> 2) the more puzzling thing (for me, at least) is that >>>> gradle.properties specifies intermine plugin version 3.0+. From >>>> what I see, these have not been uploaded to jcenter (2.1.1 seems to >>>> be the latest). >>>> >>>> Or am I doing something wrong here? >>>> >>>> Joe >>>> >>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine >>>>> for travis so I didn't catch that -- but that makes sense to use >>>>> testmine on not-localhost. >>>>> >>>>> 2. Thanks for the fix! Not sure why you are seeing this error and >>>>> we aren't though. >>>>> >>>>> On 20/10/18 00:54, Joe Carlson wrote: >>>>>> Hello, >>>>>> >>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of >>>>>> things have popped up when I tried your testmine demo: >>>>>> >>>>>> 1) The testmine demo does not appear to use anything other than >>>>>> localhost for the db server. There are a couple edits needed in >>>>>> setup.sh and testmodel.properties: >>>>>> >>>>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>>>>>> b/testmine/dbmodel/resources/testmodel.properties >>>>>>> index e3ae168..488bcd2 100644 >>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>>>> @@ -1,9 +1,9 @@ >>>>>>> -db.userprofile-test.datasource.serverName=localhost >>>>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>>>> >>>>>>> -db.unittest.datasource.serverName=localhost >>>>>>> +db.unittest.datasource.serverName=SERVER >>>>>>> db.unittest.datasource.databaseName=intermine-demo >>>>>>> db.unittest.datasource.user=PSQL_USER >>>>>>> db.unittest.datasource.password=PSQL_PWD >>>>>>> testmine> git diff setup.sh >>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>>>> index 0d13f74..41bee60 100755 >>>>>>> --- a/testmine/setup.sh >>>>>>> +++ b/testmine/setup.sh >>>>>>> @@ -74,11 +74,11 @@ fi >>>>>>> >>>>>>> echo "------> Checking databases..." >>>>>>> for db in $USERPROFILEDB $PRODDB; do >>>>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>>>> echo $db exists. >>>>>>> else >>>>>>> echo Creating $db >>>>>>> - createdb $db >>>>>>> + createdb -h $SERVER -E UTF8 $db >>>>>>> fi >>>>>>> done >>>>>> >>>>>> (you want a pull request?) >>>>>> >>>>>> 2) The other thing is that the build tripping over UTF-8 >>>>>> characters. There are a couple in the source comments (those are >>>>>> trivial), but the more important one was that I got an 'invalid >>>>>> byte sequence error' when inserting names; I assume it's Robert?? >>>>>> Setting the environment variable >>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Joe >>>>>> >>>>>> _______________________________________________ >>>>>> dev mailing list >>>>>> [hidden email] >>>>>> https://lists.intermine.org/mailman/listinfo/dev >>>> dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
Thanks Joe! Really valuable feedback.
I've updated the docs to reflect your comments, the instructions are clearer now. On 13/11/2018 17:12, Joe Carlson wrote: > Hi Julie, > > I think I see what my issues were. I was a bit confused by the docs. > > One issue was that the > https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script > say make_source is invoked as ./make_source. But this script requires > the skeletons directory below it and as far as I can tell needs to be > invokes as /path/to/intermine-scripts/make_source. > > The second issue was that > https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file > refers to bio/sources/build.gradle. I was interpreting this as a path > and kept looking for the sources folder inside bio. Perhaps this should > be labeled as bio-sources or MYMINE-bio-sources, or something. > > And the final thing is that it seems that bioSourceDBModelConfig > specific needs to be within the subprojects section of build.gradle. It > I put it at top level, I get the 'Could not find method > bioSourceDBModelConfig() for arguments...' error. > > Thanks, > > Joe > > On 11/13/18 1:06 AM, Julie Sullivan wrote: >> Hi Joe >> >> The top level of your phytomine's bio/sources directory is the correct >> placement for that additions file, and that parameter! However the >> docs were out of date, I renamed that parameter to be: >> >> globalAdditionsFile >> >> because I thought it would be less confusing! :( I've now updated the >> docs, apologies. >> >> You can look at flymine here: >> >> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 >> >> >> globalAdditionsFile = "flymine-global_additions.xml" >> >> And that file (flymine-global_additions.xml) lives in the root >> directory of flymine-bio-sources. >> >> Let me know if you have any more issues! >> Julie >> >> >> On 13/11/2018 01:49, Joe Carlson wrote: >>> Hi Julie, >>> >>> I'm still migrating to gradle. I have a question about the global >>> additions to the data model. >>> >>> You have this in the docs: >>> >>> >>> Global Additions File >>> >>> Previously the data model was merged from all data sources’ additions >>> XML file (plus the SO terms, core and genomic additons). This is no >>> longer true. Since each source is in its own JAR now, the data model >>> is self-contained for each data source. Therefore if you reference a >>> class in your data parser, it must be present in the additions file. >>> Alternatively, you can specify a single data model file that will be >>> merged into each source: >>> >>> //[in bio/sources/build.gradle] >>> // uncomment to specify an extra additions filefor your bio-sources >>> // this file will be merged with the additions filefor each datasource >>> // and included in eachsource JAR. >>> //bioSourceDBModelConfig{ >>> //extraAdditionsFile = "MY-MINE_additions.xml" >>> //} >>> >>> >>> >>> which build.gradle file are you referring to? I have a directory >>> phytomine/ from the make_mine script; I assumed it was in this >>> build.gradle file (there is not a bio/sources/ directory below >>> this.), but trying to build the db was giving the message "Could not >>> find method bioSourceDBModelConfig()". Also tried to put it in the >>> top level of my phytomine-bio-sources. Same story. >>> >>> Thanks, >>> >>> Joe >>> >>> On 11/5/18 12:17 AM, Julie Sullivan wrote: >>>> Hi Joe! >>>> >>>> 1. Yes, this is on purpose (for Travis) but you can have a local >>>> biotestmine.properties that will be used instead of the default: >>>> >>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 >>>> >>>> So the default properties file has localhost, but you can use your >>>> own properties file if you need different settings. Is that >>>> sufficient? Again, this script was written for travis but I can update. >>>> >>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got >>>> overzealous. You are right, 2.1.1 is the latest release. >>>> >>>> You are not doing anything wrong! :) >>>> Julie >>>> >>>> On 02/11/2018 22:25, Joe Carlson wrote: >>>>> Hi Julie, >>>>> >>>>> I'm back looking at the gradle build. A couple other things popped up: >>>>> >>>>> 1) setup.sh in biotestmine is also not coded for a non-localhost >>>>> postgres server. >>>>> >>>>> 2) the more puzzling thing (for me, at least) is that >>>>> gradle.properties specifies intermine plugin version 3.0+. From >>>>> what I see, these have not been uploaded to jcenter (2.1.1 seems to >>>>> be the latest). >>>>> >>>>> Or am I doing something wrong here? >>>>> >>>>> Joe >>>>> >>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine >>>>>> for travis so I didn't catch that -- but that makes sense to use >>>>>> testmine on not-localhost. >>>>>> >>>>>> 2. Thanks for the fix! Not sure why you are seeing this error and >>>>>> we aren't though. >>>>>> >>>>>> On 20/10/18 00:54, Joe Carlson wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of >>>>>>> things have popped up when I tried your testmine demo: >>>>>>> >>>>>>> 1) The testmine demo does not appear to use anything other than >>>>>>> localhost for the db server. There are a couple edits needed in >>>>>>> setup.sh and testmodel.properties: >>>>>>> >>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>>>>>>> b/testmine/dbmodel/resources/testmodel.properties >>>>>>>> index e3ae168..488bcd2 100644 >>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>>>>> @@ -1,9 +1,9 @@ >>>>>>>> -db.userprofile-test.datasource.serverName=localhost >>>>>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>>>>> >>>>>>>> -db.unittest.datasource.serverName=localhost >>>>>>>> +db.unittest.datasource.serverName=SERVER >>>>>>>> db.unittest.datasource.databaseName=intermine-demo >>>>>>>> db.unittest.datasource.user=PSQL_USER >>>>>>>> db.unittest.datasource.password=PSQL_PWD >>>>>>>> testmine> git diff setup.sh >>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>>>>> index 0d13f74..41bee60 100755 >>>>>>>> --- a/testmine/setup.sh >>>>>>>> +++ b/testmine/setup.sh >>>>>>>> @@ -74,11 +74,11 @@ fi >>>>>>>> >>>>>>>> echo "------> Checking databases..." >>>>>>>> for db in $USERPROFILEDB $PRODDB; do >>>>>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>>>>> echo $db exists. >>>>>>>> else >>>>>>>> echo Creating $db >>>>>>>> - createdb $db >>>>>>>> + createdb -h $SERVER -E UTF8 $db >>>>>>>> fi >>>>>>>> done >>>>>>> >>>>>>> (you want a pull request?) >>>>>>> >>>>>>> 2) The other thing is that the build tripping over UTF-8 >>>>>>> characters. There are a couple in the source comments (those are >>>>>>> trivial), but the more important one was that I got an 'invalid >>>>>>> byte sequence error' when inserting names; I assume it's Robert?? >>>>>>> Setting the environment variable >>>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Joe >>>>>>> >>>>>>> _______________________________________________ >>>>>>> dev mailing list >>>>>>> [hidden email] >>>>>>> https://lists.intermine.org/mailman/listinfo/dev >>>>> dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
Thanks for the updates.
Another thing I noticed is a problem with the docs which has reared its ugly head again. Type ‘merging’ into the search box. The first two links I get are ‘Model Merging’ and ‘Data Model Overview’. These links work properly. But the third link ‘Database’ goes to a ‘SORRY This page does not exist yet’ error page. The issue seems to be that links that go to /path/to/page/index do not work, but /path/to/page/anything_else are OK. And, if you take /path/to/page/index and append .html to the URL, all is good. Joe > On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]> wrote: > > Thanks Joe! Really valuable feedback. > > I've updated the docs to reflect your comments, the instructions are clearer now. > > On 13/11/2018 17:12, Joe Carlson wrote: >> Hi Julie, >> I think I see what my issues were. I was a bit confused by the docs. >> One issue was that the https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script say make_source is invoked as ./make_source. But this script requires the skeletons directory below it and as far as I can tell needs to be invokes as /path/to/intermine-scripts/make_source. >> The second issue was that https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file refers to bio/sources/build.gradle. I was interpreting this as a path and kept looking for the sources folder inside bio. Perhaps this should be labeled as bio-sources or MYMINE-bio-sources, or something. >> And the final thing is that it seems that bioSourceDBModelConfig specific needs to be within the subprojects section of build.gradle. It I put it at top level, I get the 'Could not find method bioSourceDBModelConfig() for arguments...' error. >> Thanks, >> Joe >> On 11/13/18 1:06 AM, Julie Sullivan wrote: >>> Hi Joe >>> >>> The top level of your phytomine's bio/sources directory is the correct placement for that additions file, and that parameter! However the docs were out of date, I renamed that parameter to be: >>> >>> globalAdditionsFile >>> >>> because I thought it would be less confusing! :( I've now updated the docs, apologies. >>> >>> You can look at flymine here: >>> >>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 >>> >>> globalAdditionsFile = "flymine-global_additions.xml" >>> >>> And that file (flymine-global_additions.xml) lives in the root directory of flymine-bio-sources. >>> >>> Let me know if you have any more issues! >>> Julie >>> >>> >>> On 13/11/2018 01:49, Joe Carlson wrote: >>>> Hi Julie, >>>> >>>> I'm still migrating to gradle. I have a question about the global additions to the data model. >>>> >>>> You have this in the docs: >>>> >>>> >>>> Global Additions File >>>> >>>> Previously the data model was merged from all data sources’ additions XML file (plus the SO terms, core and genomic additons). This is no longer true. Since each source is in its own JAR now, the data model is self-contained for each data source. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source: >>>> >>>> //[in bio/sources/build.gradle] >>>> // uncomment to specify an extra additions filefor your bio-sources >>>> // this file will be merged with the additions filefor each datasource >>>> // and included in eachsource JAR. >>>> //bioSourceDBModelConfig{ >>>> //extraAdditionsFile = "MY-MINE_additions.xml" >>>> //} >>>> >>>> >>>> >>>> which build.gradle file are you referring to? I have a directory phytomine/ from the make_mine script; I assumed it was in this build.gradle file (there is not a bio/sources/ directory below this.), but trying to build the db was giving the message "Could not find method bioSourceDBModelConfig()". Also tried to put it in the top level of my phytomine-bio-sources. Same story. >>>> >>>> Thanks, >>>> >>>> Joe >>>> >>>> On 11/5/18 12:17 AM, Julie Sullivan wrote: >>>>> Hi Joe! >>>>> >>>>> 1. Yes, this is on purpose (for Travis) but you can have a local biotestmine.properties that will be used instead of the default: >>>>> >>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 >>>>> >>>>> So the default properties file has localhost, but you can use your own properties file if you need different settings. Is that sufficient? Again, this script was written for travis but I can update. >>>>> >>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got overzealous. You are right, 2.1.1 is the latest release. >>>>> >>>>> You are not doing anything wrong! :) >>>>> Julie >>>>> >>>>> On 02/11/2018 22:25, Joe Carlson wrote: >>>>>> Hi Julie, >>>>>> >>>>>> I'm back looking at the gradle build. A couple other things popped up: >>>>>> >>>>>> 1) setup.sh in biotestmine is also not coded for a non-localhost postgres server. >>>>>> >>>>>> 2) the more puzzling thing (for me, at least) is that gradle.properties specifies intermine plugin version 3.0+. From what I see, these have not been uploaded to jcenter (2.1.1 seems to be the latest). >>>>>> >>>>>> Or am I doing something wrong here? >>>>>> >>>>>> Joe >>>>>> >>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for travis so I didn't catch that -- but that makes sense to use testmine on not-localhost. >>>>>>> >>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error and we aren't though. >>>>>>> >>>>>>> On 20/10/18 00:54, Joe Carlson wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of things have popped up when I tried your testmine demo: >>>>>>>> >>>>>>>> 1) The testmine demo does not appear to use anything other than localhost for the db server. There are a couple edits needed in setup.sh and testmodel.properties: >>>>>>>> >>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>> index e3ae168..488bcd2 100644 >>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>> @@ -1,9 +1,9 @@ >>>>>>>>> -db.userprofile-test.datasource.serverName=localhost >>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>>>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>>>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>>>>>> >>>>>>>>> -db.unittest.datasource.serverName=localhost >>>>>>>>> +db.unittest.datasource.serverName=SERVER >>>>>>>>> db.unittest.datasource.databaseName=intermine-demo >>>>>>>>> db.unittest.datasource.user=PSQL_USER >>>>>>>>> db.unittest.datasource.password=PSQL_PWD >>>>>>>>> testmine> git diff setup.sh >>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>>>>>> index 0d13f74..41bee60 100755 >>>>>>>>> --- a/testmine/setup.sh >>>>>>>>> +++ b/testmine/setup.sh >>>>>>>>> @@ -74,11 +74,11 @@ fi >>>>>>>>> >>>>>>>>> echo "------> Checking databases..." >>>>>>>>> for db in $USERPROFILEDB $PRODDB; do >>>>>>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>>>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>>>>>> echo $db exists. >>>>>>>>> else >>>>>>>>> echo Creating $db >>>>>>>>> - createdb $db >>>>>>>>> + createdb -h $SERVER -E UTF8 $db >>>>>>>>> fi >>>>>>>>> done >>>>>>>> >>>>>>>> (you want a pull request?) >>>>>>>> >>>>>>>> 2) The other thing is that the build tripping over UTF-8 characters. There are a couple in the source comments (those are trivial), but the more important one was that I got an 'invalid byte sequence error' when inserting names; I assume it's Robert?? Setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Joe >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> dev mailing list >>>>>>>> [hidden email] >>>>>>>> https://lists.intermine.org/mailman/listinfo/dev >>>>>> _______________________________________________ dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
Yes! I have seen this too, but can't really figure out what's going on:
https://github.com/intermine/intermine/issues/1497 I asked the ReadTheDocs folks and they said it might be the stylesheet we're using as they couldn't replicate, so I'll try updating that next. On 14/11/2018 18:33, Joe Carlson wrote: > Thanks for the updates. > > Another thing I noticed is a problem with the docs which has reared its ugly head again. > > Type ‘merging’ into the search box. The first two links I get are ‘Model Merging’ and ‘Data Model Overview’. These links work properly. But the third link ‘Database’ goes to a ‘SORRY This page does not exist yet’ error page. > > The issue seems to be that links that go to /path/to/page/index do not work, but /path/to/page/anything_else are OK. And, if you take /path/to/page/index and append .html to the URL, all is good. > > Joe > >> On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]> wrote: >> >> Thanks Joe! Really valuable feedback. >> >> I've updated the docs to reflect your comments, the instructions are clearer now. >> >> On 13/11/2018 17:12, Joe Carlson wrote: >>> Hi Julie, >>> I think I see what my issues were. I was a bit confused by the docs. >>> One issue was that the https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script say make_source is invoked as ./make_source. But this script requires the skeletons directory below it and as far as I can tell needs to be invokes as /path/to/intermine-scripts/make_source. >>> The second issue was that https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file refers to bio/sources/build.gradle. I was interpreting this as a path and kept looking for the sources folder inside bio. Perhaps this should be labeled as bio-sources or MYMINE-bio-sources, or something. >>> And the final thing is that it seems that bioSourceDBModelConfig specific needs to be within the subprojects section of build.gradle. It I put it at top level, I get the 'Could not find method bioSourceDBModelConfig() for arguments...' error. >>> Thanks, >>> Joe >>> On 11/13/18 1:06 AM, Julie Sullivan wrote: >>>> Hi Joe >>>> >>>> The top level of your phytomine's bio/sources directory is the correct placement for that additions file, and that parameter! However the docs were out of date, I renamed that parameter to be: >>>> >>>> globalAdditionsFile >>>> >>>> because I thought it would be less confusing! :( I've now updated the docs, apologies. >>>> >>>> You can look at flymine here: >>>> >>>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 >>>> >>>> globalAdditionsFile = "flymine-global_additions.xml" >>>> >>>> And that file (flymine-global_additions.xml) lives in the root directory of flymine-bio-sources. >>>> >>>> Let me know if you have any more issues! >>>> Julie >>>> >>>> >>>> On 13/11/2018 01:49, Joe Carlson wrote: >>>>> Hi Julie, >>>>> >>>>> I'm still migrating to gradle. I have a question about the global additions to the data model. >>>>> >>>>> You have this in the docs: >>>>> >>>>> >>>>> Global Additions File >>>>> >>>>> Previously the data model was merged from all data sources’ additions XML file (plus the SO terms, core and genomic additons). This is no longer true. Since each source is in its own JAR now, the data model is self-contained for each data source. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source: >>>>> >>>>> //[in bio/sources/build.gradle] >>>>> // uncomment to specify an extra additions filefor your bio-sources >>>>> // this file will be merged with the additions filefor each datasource >>>>> // and included in eachsource JAR. >>>>> //bioSourceDBModelConfig{ >>>>> //extraAdditionsFile = "MY-MINE_additions.xml" >>>>> //} >>>>> >>>>> >>>>> >>>>> which build.gradle file are you referring to? I have a directory phytomine/ from the make_mine script; I assumed it was in this build.gradle file (there is not a bio/sources/ directory below this.), but trying to build the db was giving the message "Could not find method bioSourceDBModelConfig()". Also tried to put it in the top level of my phytomine-bio-sources. Same story. >>>>> >>>>> Thanks, >>>>> >>>>> Joe >>>>> >>>>> On 11/5/18 12:17 AM, Julie Sullivan wrote: >>>>>> Hi Joe! >>>>>> >>>>>> 1. Yes, this is on purpose (for Travis) but you can have a local biotestmine.properties that will be used instead of the default: >>>>>> >>>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 >>>>>> >>>>>> So the default properties file has localhost, but you can use your own properties file if you need different settings. Is that sufficient? Again, this script was written for travis but I can update. >>>>>> >>>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got overzealous. You are right, 2.1.1 is the latest release. >>>>>> >>>>>> You are not doing anything wrong! :) >>>>>> Julie >>>>>> >>>>>> On 02/11/2018 22:25, Joe Carlson wrote: >>>>>>> Hi Julie, >>>>>>> >>>>>>> I'm back looking at the gradle build. A couple other things popped up: >>>>>>> >>>>>>> 1) setup.sh in biotestmine is also not coded for a non-localhost postgres server. >>>>>>> >>>>>>> 2) the more puzzling thing (for me, at least) is that gradle.properties specifies intermine plugin version 3.0+. From what I see, these have not been uploaded to jcenter (2.1.1 seems to be the latest). >>>>>>> >>>>>>> Or am I doing something wrong here? >>>>>>> >>>>>>> Joe >>>>>>> >>>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for travis so I didn't catch that -- but that makes sense to use testmine on not-localhost. >>>>>>>> >>>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error and we aren't though. >>>>>>>> >>>>>>>> On 20/10/18 00:54, Joe Carlson wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of things have popped up when I tried your testmine demo: >>>>>>>>> >>>>>>>>> 1) The testmine demo does not appear to use anything other than localhost for the db server. There are a couple edits needed in setup.sh and testmodel.properties: >>>>>>>>> >>>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>> index e3ae168..488bcd2 100644 >>>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>> @@ -1,9 +1,9 @@ >>>>>>>>>> -db.userprofile-test.datasource.serverName=localhost >>>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>>>>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>>>>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>>>>>>> >>>>>>>>>> -db.unittest.datasource.serverName=localhost >>>>>>>>>> +db.unittest.datasource.serverName=SERVER >>>>>>>>>> db.unittest.datasource.databaseName=intermine-demo >>>>>>>>>> db.unittest.datasource.user=PSQL_USER >>>>>>>>>> db.unittest.datasource.password=PSQL_PWD >>>>>>>>>> testmine> git diff setup.sh >>>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>>>>>>> index 0d13f74..41bee60 100755 >>>>>>>>>> --- a/testmine/setup.sh >>>>>>>>>> +++ b/testmine/setup.sh >>>>>>>>>> @@ -74,11 +74,11 @@ fi >>>>>>>>>> >>>>>>>>>> echo "------> Checking databases..." >>>>>>>>>> for db in $USERPROFILEDB $PRODDB; do >>>>>>>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>>>>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>>>>>>> echo $db exists. >>>>>>>>>> else >>>>>>>>>> echo Creating $db >>>>>>>>>> - createdb $db >>>>>>>>>> + createdb -h $SERVER -E UTF8 $db >>>>>>>>>> fi >>>>>>>>>> done >>>>>>>>> >>>>>>>>> (you want a pull request?) >>>>>>>>> >>>>>>>>> 2) The other thing is that the build tripping over UTF-8 characters. There are a couple in the source comments (those are trivial), but the more important one was that I got an 'invalid byte sequence error' when inserting names; I assume it's Robert?? Setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Joe >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> dev mailing list >>>>>>>>> [hidden email] >>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev >>>>>>> > dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
yeah. that's weird. I remember seeing this in the past, but thought it
had been fixed. It looks like the non-index pages are generating responding with a 301 'moved permanently' response and appending a / to the URL > > curl -v --head > "https://intermine.readthedocs.io/en/latest/database/data-sources/library/so" > * Trying 137.116.78.48... > <snip> > > HEAD /en/latest/database/data-sources/library/so HTTP/1.1 > > Host: intermine.readthedocs.io > > User-Agent: curl/7.60.0 > > Accept: */* > > > < HTTP/1.1 301 Moved Permanently > HTTP/1.1 301 Moved Permanently > < Content-Length: 194 > Content-Length: 194 > < Content-Type: text/html > Content-Type: text/html > < Location: > http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/ > Location: > http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/ > < Server: nginx/1.14.0 (Ubuntu) > Server: nginx/1.14.0 (Ubuntu) > < X-Subdomain-TryFiles: True > X-Subdomain-TryFiles: True > < X-Served: Nginx > X-Served: Nginx > < X-Deity: web03 > X-Deity: web03 > < Date: Thu, 15 Nov 2018 16:44:46 GMT > Date: Thu, 15 Nov 2018 16:44:46 GMT > > < > * Connection #0 to host intermine.readthedocs.io left intact But URLs that point to /index respond with a 404 without any redirect. Are you running nginx as a proxy server or is it delivering content? It looks like the rewrite rules for index URL's needs a little tweaking. Joe >> Thanks for the updates. >> >> Another thing I noticed is a problem with the docs which has reared >> its ugly head again. >> >> Type ‘merging’ into the search box. The first two links I get are >> ‘Model Merging’ and ‘Data Model Overview’. These links work properly. >> But the third link ‘Database’ goes to a ‘SORRY This page does not >> exist yet’ error page. >> >> The issue seems to be that links that go to /path/to/page/index do >> not work, but /path/to/page/anything_else are OK. And, if you take >> /path/to/page/index and append .html to the URL, all is good. >> >> Joe >> >>> On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]> >>> wrote: >>> >>> Thanks Joe! Really valuable feedback. >>> >>> I've updated the docs to reflect your comments, the instructions are >>> clearer now. >>> >>> On 13/11/2018 17:12, Joe Carlson wrote: >>>> Hi Julie, >>>> I think I see what my issues were. I was a bit confused by the docs. >>>> One issue was that the >>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script >>>> say make_source is invoked as ./make_source. But this script >>>> requires the skeletons directory below it and as far as I can tell >>>> needs to be invokes as /path/to/intermine-scripts/make_source. >>>> The second issue was that >>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file >>>> refers to bio/sources/build.gradle. I was interpreting this as a >>>> path and kept looking for the sources folder inside bio. Perhaps >>>> this should be labeled as bio-sources or MYMINE-bio-sources, or >>>> something. >>>> And the final thing is that it seems that bioSourceDBModelConfig >>>> specific needs to be within the subprojects section of >>>> build.gradle. It I put it at top level, I get the 'Could not find >>>> method bioSourceDBModelConfig() for arguments...' error. >>>> Thanks, >>>> Joe >>>> On 11/13/18 1:06 AM, Julie Sullivan wrote: >>>>> Hi Joe >>>>> >>>>> The top level of your phytomine's bio/sources directory is the >>>>> correct placement for that additions file, and that parameter! >>>>> However the docs were out of date, I renamed that parameter to be: >>>>> >>>>> globalAdditionsFile >>>>> >>>>> because I thought it would be less confusing! :( I've now updated >>>>> the docs, apologies. >>>>> >>>>> You can look at flymine here: >>>>> >>>>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 >>>>> >>>>> >>>>> globalAdditionsFile = "flymine-global_additions.xml" >>>>> >>>>> And that file (flymine-global_additions.xml) lives in the root >>>>> directory of flymine-bio-sources. >>>>> >>>>> Let me know if you have any more issues! >>>>> Julie >>>>> >>>>> >>>>> On 13/11/2018 01:49, Joe Carlson wrote: >>>>>> Hi Julie, >>>>>> >>>>>> I'm still migrating to gradle. I have a question about the global >>>>>> additions to the data model. >>>>>> >>>>>> You have this in the docs: >>>>>> >>>>>> >>>>>> Global Additions File >>>>>> >>>>>> Previously the data model was merged from all data sources’ >>>>>> additions XML file (plus the SO terms, core and genomic >>>>>> additons). This is no longer true. Since each source is in its >>>>>> own JAR now, the data model is self-contained for each data >>>>>> source. Therefore if you reference a class in your data parser, >>>>>> it must be present in the additions file. Alternatively, you can >>>>>> specify a single data model file that will be merged into each >>>>>> source: >>>>>> >>>>>> //[in bio/sources/build.gradle] >>>>>> // uncomment to specify an extra additions filefor your bio-sources >>>>>> // this file will be merged with the additions filefor each >>>>>> datasource >>>>>> // and included in eachsource JAR. >>>>>> //bioSourceDBModelConfig{ >>>>>> //extraAdditionsFile = "MY-MINE_additions.xml" >>>>>> //} >>>>>> >>>>>> >>>>>> >>>>>> which build.gradle file are you referring to? I have a directory >>>>>> phytomine/ from the make_mine script; I assumed it was in this >>>>>> build.gradle file (there is not a bio/sources/ directory below >>>>>> this.), but trying to build the db was giving the message "Could >>>>>> not find method bioSourceDBModelConfig()". Also tried to put it >>>>>> in the top level of my phytomine-bio-sources. Same story. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Joe >>>>>> >>>>>> On 11/5/18 12:17 AM, Julie Sullivan wrote: >>>>>>> Hi Joe! >>>>>>> >>>>>>> 1. Yes, this is on purpose (for Travis) but you can have a local >>>>>>> biotestmine.properties that will be used instead of the default: >>>>>>> >>>>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 >>>>>>> >>>>>>> So the default properties file has localhost, but you can use >>>>>>> your own properties file if you need different settings. Is that >>>>>>> sufficient? Again, this script was written for travis but I can >>>>>>> update. >>>>>>> >>>>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got >>>>>>> overzealous. You are right, 2.1.1 is the latest release. >>>>>>> >>>>>>> You are not doing anything wrong! :) >>>>>>> Julie >>>>>>> >>>>>>> On 02/11/2018 22:25, Joe Carlson wrote: >>>>>>>> Hi Julie, >>>>>>>> >>>>>>>> I'm back looking at the gradle build. A couple other things >>>>>>>> popped up: >>>>>>>> >>>>>>>> 1) setup.sh in biotestmine is also not coded for a >>>>>>>> non-localhost postgres server. >>>>>>>> >>>>>>>> 2) the more puzzling thing (for me, at least) is that >>>>>>>> gradle.properties specifies intermine plugin version 3.0+. From >>>>>>>> what I see, these have not been uploaded to jcenter (2.1.1 >>>>>>>> seems to be the latest). >>>>>>>> >>>>>>>> Or am I doing something wrong here? >>>>>>>> >>>>>>>> Joe >>>>>>>> >>>>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>>>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use >>>>>>>>> testmine for travis so I didn't catch that -- but that makes >>>>>>>>> sense to use testmine on not-localhost. >>>>>>>>> >>>>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error >>>>>>>>> and we aren't though. >>>>>>>>> >>>>>>>>> On 20/10/18 00:54, Joe Carlson wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of >>>>>>>>>> things have popped up when I tried your testmine demo: >>>>>>>>>> >>>>>>>>>> 1) The testmine demo does not appear to use anything other >>>>>>>>>> than localhost for the db server. There are a couple edits >>>>>>>>>> needed in setup.sh and testmodel.properties: >>>>>>>>>> >>>>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>> b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>> index e3ae168..488bcd2 100644 >>>>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>> @@ -1,9 +1,9 @@ >>>>>>>>>>> -db.userprofile-test.datasource.serverName=localhost >>>>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>>>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>>>>>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>>>>>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>>>>>>>> >>>>>>>>>>> -db.unittest.datasource.serverName=localhost >>>>>>>>>>> +db.unittest.datasource.serverName=SERVER >>>>>>>>>>> db.unittest.datasource.databaseName=intermine-demo >>>>>>>>>>> db.unittest.datasource.user=PSQL_USER >>>>>>>>>>> db.unittest.datasource.password=PSQL_PWD >>>>>>>>>>> testmine> git diff setup.sh >>>>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>>>>>>>> index 0d13f74..41bee60 100755 >>>>>>>>>>> --- a/testmine/setup.sh >>>>>>>>>>> +++ b/testmine/setup.sh >>>>>>>>>>> @@ -74,11 +74,11 @@ fi >>>>>>>>>>> >>>>>>>>>>> echo "------> Checking databases..." >>>>>>>>>>> for db in $USERPROFILEDB $PRODDB; do >>>>>>>>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>>>>>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>>>>>>>> echo $db exists. >>>>>>>>>>> else >>>>>>>>>>> echo Creating $db >>>>>>>>>>> - createdb $db >>>>>>>>>>> + createdb -h $SERVER -E UTF8 $db >>>>>>>>>>> fi >>>>>>>>>>> done >>>>>>>>>> >>>>>>>>>> (you want a pull request?) >>>>>>>>>> >>>>>>>>>> 2) The other thing is that the build tripping over UTF-8 >>>>>>>>>> characters. There are a couple in the source comments (those >>>>>>>>>> are trivial), but the more important one was that I got an >>>>>>>>>> 'invalid byte sequence error' when inserting names; I assume >>>>>>>>>> it's Robert?? Setting the environment variable >>>>>>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed >>>>>>>>>> that. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Joe >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> dev mailing list >>>>>>>>>> [hidden email] >>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev >>>>>>>> >> dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
Re: Starting on InterMine 2.0....
|
|
That's what I thought! But they are hosting, and on that ticket the read
the docs dev commented they couldn't replicate. And blamed it on our beautiful, innocent stylesheet. On 15/11/2018 16:50, Joe Carlson wrote: > yeah. that's weird. I remember seeing this in the past, but thought it > had been fixed. > > It looks like the non-index pages are generating responding with a 301 > 'moved permanently' response and appending a / to the URL > >> > curl -v --head >> "https://intermine.readthedocs.io/en/latest/database/data-sources/library/so" >> >> * Trying 137.116.78.48... >> <snip> >> > HEAD /en/latest/database/data-sources/library/so HTTP/1.1 >> > Host: intermine.readthedocs.io >> > User-Agent: curl/7.60.0 >> > Accept: */* >> > >> < HTTP/1.1 301 Moved Permanently >> HTTP/1.1 301 Moved Permanently >> < Content-Length: 194 >> Content-Length: 194 >> < Content-Type: text/html >> Content-Type: text/html >> < Location: >> http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/ >> >> Location: >> http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/ >> >> < Server: nginx/1.14.0 (Ubuntu) >> Server: nginx/1.14.0 (Ubuntu) >> < X-Subdomain-TryFiles: True >> X-Subdomain-TryFiles: True >> < X-Served: Nginx >> X-Served: Nginx >> < X-Deity: web03 >> X-Deity: web03 >> < Date: Thu, 15 Nov 2018 16:44:46 GMT >> Date: Thu, 15 Nov 2018 16:44:46 GMT >> >> < >> * Connection #0 to host intermine.readthedocs.io left intact > > But URLs that point to /index respond with a 404 without any redirect. > Are you running nginx as a proxy server or is it delivering content? It > looks like the rewrite rules for index URL's needs a little tweaking. > > > Joe > >>> Thanks for the updates. >>> >>> Another thing I noticed is a problem with the docs which has reared >>> its ugly head again. >>> >>> Type ‘merging’ into the search box. The first two links I get are >>> ‘Model Merging’ and ‘Data Model Overview’. These links work properly. >>> But the third link ‘Database’ goes to a ‘SORRY This page does not >>> exist yet’ error page. >>> >>> The issue seems to be that links that go to /path/to/page/index do >>> not work, but /path/to/page/anything_else are OK. And, if you take >>> /path/to/page/index and append .html to the URL, all is good. >>> >>> Joe >>> >>>> On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]> >>>> wrote: >>>> >>>> Thanks Joe! Really valuable feedback. >>>> >>>> I've updated the docs to reflect your comments, the instructions are >>>> clearer now. >>>> >>>> On 13/11/2018 17:12, Joe Carlson wrote: >>>>> Hi Julie, >>>>> I think I see what my issues were. I was a bit confused by the docs. >>>>> One issue was that the >>>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script >>>>> say make_source is invoked as ./make_source. But this script >>>>> requires the skeletons directory below it and as far as I can tell >>>>> needs to be invokes as /path/to/intermine-scripts/make_source. >>>>> The second issue was that >>>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file >>>>> refers to bio/sources/build.gradle. I was interpreting this as a >>>>> path and kept looking for the sources folder inside bio. Perhaps >>>>> this should be labeled as bio-sources or MYMINE-bio-sources, or >>>>> something. >>>>> And the final thing is that it seems that bioSourceDBModelConfig >>>>> specific needs to be within the subprojects section of >>>>> build.gradle. It I put it at top level, I get the 'Could not find >>>>> method bioSourceDBModelConfig() for arguments...' error. >>>>> Thanks, >>>>> Joe >>>>> On 11/13/18 1:06 AM, Julie Sullivan wrote: >>>>>> Hi Joe >>>>>> >>>>>> The top level of your phytomine's bio/sources directory is the >>>>>> correct placement for that additions file, and that parameter! >>>>>> However the docs were out of date, I renamed that parameter to be: >>>>>> >>>>>> globalAdditionsFile >>>>>> >>>>>> because I thought it would be less confusing! :( I've now updated >>>>>> the docs, apologies. >>>>>> >>>>>> You can look at flymine here: >>>>>> >>>>>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 >>>>>> >>>>>> >>>>>> globalAdditionsFile = "flymine-global_additions.xml" >>>>>> >>>>>> And that file (flymine-global_additions.xml) lives in the root >>>>>> directory of flymine-bio-sources. >>>>>> >>>>>> Let me know if you have any more issues! >>>>>> Julie >>>>>> >>>>>> >>>>>> On 13/11/2018 01:49, Joe Carlson wrote: >>>>>>> Hi Julie, >>>>>>> >>>>>>> I'm still migrating to gradle. I have a question about the global >>>>>>> additions to the data model. >>>>>>> >>>>>>> You have this in the docs: >>>>>>> >>>>>>> >>>>>>> Global Additions File >>>>>>> >>>>>>> Previously the data model was merged from all data sources’ >>>>>>> additions XML file (plus the SO terms, core and genomic >>>>>>> additons). This is no longer true. Since each source is in its >>>>>>> own JAR now, the data model is self-contained for each data >>>>>>> source. Therefore if you reference a class in your data parser, >>>>>>> it must be present in the additions file. Alternatively, you can >>>>>>> specify a single data model file that will be merged into each >>>>>>> source: >>>>>>> >>>>>>> //[in bio/sources/build.gradle] >>>>>>> // uncomment to specify an extra additions filefor your bio-sources >>>>>>> // this file will be merged with the additions filefor each >>>>>>> datasource >>>>>>> // and included in eachsource JAR. >>>>>>> //bioSourceDBModelConfig{ >>>>>>> //extraAdditionsFile = "MY-MINE_additions.xml" >>>>>>> //} >>>>>>> >>>>>>> >>>>>>> >>>>>>> which build.gradle file are you referring to? I have a directory >>>>>>> phytomine/ from the make_mine script; I assumed it was in this >>>>>>> build.gradle file (there is not a bio/sources/ directory below >>>>>>> this.), but trying to build the db was giving the message "Could >>>>>>> not find method bioSourceDBModelConfig()". Also tried to put it >>>>>>> in the top level of my phytomine-bio-sources. Same story. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Joe >>>>>>> >>>>>>> On 11/5/18 12:17 AM, Julie Sullivan wrote: >>>>>>>> Hi Joe! >>>>>>>> >>>>>>>> 1. Yes, this is on purpose (for Travis) but you can have a local >>>>>>>> biotestmine.properties that will be used instead of the default: >>>>>>>> >>>>>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80 >>>>>>>> >>>>>>>> So the default properties file has localhost, but you can use >>>>>>>> your own properties file if you need different settings. Is that >>>>>>>> sufficient? Again, this script was written for travis but I can >>>>>>>> update. >>>>>>>> >>>>>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got >>>>>>>> overzealous. You are right, 2.1.1 is the latest release. >>>>>>>> >>>>>>>> You are not doing anything wrong! :) >>>>>>>> Julie >>>>>>>> >>>>>>>> On 02/11/2018 22:25, Joe Carlson wrote: >>>>>>>>> Hi Julie, >>>>>>>>> >>>>>>>>> I'm back looking at the gradle build. A couple other things >>>>>>>>> popped up: >>>>>>>>> >>>>>>>>> 1) setup.sh in biotestmine is also not coded for a >>>>>>>>> non-localhost postgres server. >>>>>>>>> >>>>>>>>> 2) the more puzzling thing (for me, at least) is that >>>>>>>>> gradle.properties specifies intermine plugin version 3.0+. From >>>>>>>>> what I see, these have not been uploaded to jcenter (2.1.1 >>>>>>>>> seems to be the latest). >>>>>>>>> >>>>>>>>> Or am I doing something wrong here? >>>>>>>>> >>>>>>>>> Joe >>>>>>>>> >>>>>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote: >>>>>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use >>>>>>>>>> testmine for travis so I didn't catch that -- but that makes >>>>>>>>>> sense to use testmine on not-localhost. >>>>>>>>>> >>>>>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error >>>>>>>>>> and we aren't though. >>>>>>>>>> >>>>>>>>>> On 20/10/18 00:54, Joe Carlson wrote: >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of >>>>>>>>>>> things have popped up when I tried your testmine demo: >>>>>>>>>>> >>>>>>>>>>> 1) The testmine demo does not appear to use anything other >>>>>>>>>>> than localhost for the db server. There are a couple edits >>>>>>>>>>> needed in setup.sh and testmodel.properties: >>>>>>>>>>> >>>>>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties >>>>>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>>> b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>>> index e3ae168..488bcd2 100644 >>>>>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties >>>>>>>>>>>> @@ -1,9 +1,9 @@ >>>>>>>>>>>> -db.userprofile-test.datasource.serverName=localhost >>>>>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER >>>>>>>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo >>>>>>>>>>>> db.userprofile-test.datasource.user=PSQL_USER >>>>>>>>>>>> db.userprofile-test.datasource.password=PSQL_PWD >>>>>>>>>>>> >>>>>>>>>>>> -db.unittest.datasource.serverName=localhost >>>>>>>>>>>> +db.unittest.datasource.serverName=SERVER >>>>>>>>>>>> db.unittest.datasource.databaseName=intermine-demo >>>>>>>>>>>> db.unittest.datasource.user=PSQL_USER >>>>>>>>>>>> db.unittest.datasource.password=PSQL_PWD >>>>>>>>>>>> testmine> git diff setup.sh >>>>>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh >>>>>>>>>>>> index 0d13f74..41bee60 100755 >>>>>>>>>>>> --- a/testmine/setup.sh >>>>>>>>>>>> +++ b/testmine/setup.sh >>>>>>>>>>>> @@ -74,11 +74,11 @@ fi >>>>>>>>>>>> >>>>>>>>>>>> echo "------> Checking databases..." >>>>>>>>>>>> for db in $USERPROFILEDB $PRODDB; do >>>>>>>>>>>> - if psql --list | egrep -q '\s'$db'\s'; then >>>>>>>>>>>> + if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then >>>>>>>>>>>> echo $db exists. >>>>>>>>>>>> else >>>>>>>>>>>> echo Creating $db >>>>>>>>>>>> - createdb $db >>>>>>>>>>>> + createdb -h $SERVER -E UTF8 $db >>>>>>>>>>>> fi >>>>>>>>>>>> done >>>>>>>>>>> >>>>>>>>>>> (you want a pull request?) >>>>>>>>>>> >>>>>>>>>>> 2) The other thing is that the build tripping over UTF-8 >>>>>>>>>>> characters. There are a couple in the source comments (those >>>>>>>>>>> are trivial), but the more important one was that I got an >>>>>>>>>>> 'invalid byte sequence error' when inserting names; I assume >>>>>>>>>>> it's Robert?? Setting the environment variable >>>>>>>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed >>>>>>>>>>> that. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Joe >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> dev mailing list >>>>>>>>>>> [hidden email] >>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev >>>>>>>>> >>> dev mailing list [hidden email] https://lists.intermine.org/mailman/listinfo/dev |
«
Return to InterMine
|
1 view|%1 views
| Free forum by Nabble | Edit this page |