Starting on InterMine 2.0....

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Starting on InterMine 2.0....

joe carlson
Hello,

I'm (finally) trying to build InterMine 2.0. But a couple of things have
popped up when I tried your testmine demo:

1) The testmine demo does not appear to use anything other than
localhost for the db server. There are a couple edits needed in setup.sh
and testmodel.properties:

> testmine> git diff dbmodel/resources/testmodel.properties
> diff --git a/testmine/dbmodel/resources/testmodel.properties
> b/testmine/dbmodel/resources/testmodel.properties
> index e3ae168..488bcd2 100644
> --- a/testmine/dbmodel/resources/testmodel.properties
> +++ b/testmine/dbmodel/resources/testmodel.properties
> @@ -1,9 +1,9 @@
> -db.userprofile-test.datasource.serverName=localhost
> +db.userprofile-test.datasource.serverName=SERVER
>  db.userprofile-test.datasource.databaseName=userprofile-demo
>  db.userprofile-test.datasource.user=PSQL_USER
>  db.userprofile-test.datasource.password=PSQL_PWD
>
> -db.unittest.datasource.serverName=localhost
> +db.unittest.datasource.serverName=SERVER
>  db.unittest.datasource.databaseName=intermine-demo
>  db.unittest.datasource.user=PSQL_USER
>  db.unittest.datasource.password=PSQL_PWD
> testmine> git diff setup.sh
> diff --git a/testmine/setup.sh b/testmine/setup.sh
> index 0d13f74..41bee60 100755
> --- a/testmine/setup.sh
> +++ b/testmine/setup.sh
> @@ -74,11 +74,11 @@ fi
>
>  echo "------> Checking databases..."
>  for db in $USERPROFILEDB $PRODDB; do
> -    if psql --list | egrep -q '\s'$db'\s'; then
> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>          echo $db exists.
>      else
>          echo Creating $db
> -        createdb $db
> +        createdb -h $SERVER -E UTF8 $db
>      fi
>  done

(you want a pull request?)

2) The other thing is that the build tripping over UTF-8 characters.
There are a couple in the source comments (those are trivial), but the
more important one was that I got an 'invalid byte sequence error' when
inserting names; I assume it's Robert?? Setting the environment variable
JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.

Thanks,

Joe

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

Julie Sullivan-2
1. Whoops, yes, thanks for that. I'll fix. We only use testmine for
travis so I didn't catch that -- but that makes sense to use testmine on
not-localhost.

2. Thanks for the fix! Not sure why you are seeing this error and we
aren't though.

On 20/10/18 00:54, Joe Carlson wrote:

> Hello,
>
> I'm (finally) trying to build InterMine 2.0. But a couple of things have
> popped up when I tried your testmine demo:
>
> 1) The testmine demo does not appear to use anything other than
> localhost for the db server. There are a couple edits needed in setup.sh
> and testmodel.properties:
>
>> testmine> git diff dbmodel/resources/testmodel.properties
>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>> b/testmine/dbmodel/resources/testmodel.properties
>> index e3ae168..488bcd2 100644
>> --- a/testmine/dbmodel/resources/testmodel.properties
>> +++ b/testmine/dbmodel/resources/testmodel.properties
>> @@ -1,9 +1,9 @@
>> -db.userprofile-test.datasource.serverName=localhost
>> +db.userprofile-test.datasource.serverName=SERVER
>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>  db.userprofile-test.datasource.user=PSQL_USER
>>  db.userprofile-test.datasource.password=PSQL_PWD
>>
>> -db.unittest.datasource.serverName=localhost
>> +db.unittest.datasource.serverName=SERVER
>>  db.unittest.datasource.databaseName=intermine-demo
>>  db.unittest.datasource.user=PSQL_USER
>>  db.unittest.datasource.password=PSQL_PWD
>> testmine> git diff setup.sh
>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>> index 0d13f74..41bee60 100755
>> --- a/testmine/setup.sh
>> +++ b/testmine/setup.sh
>> @@ -74,11 +74,11 @@ fi
>>
>>  echo "------> Checking databases..."
>>  for db in $USERPROFILEDB $PRODDB; do
>> -    if psql --list | egrep -q '\s'$db'\s'; then
>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>          echo $db exists.
>>      else
>>          echo Creating $db
>> -        createdb $db
>> +        createdb -h $SERVER -E UTF8 $db
>>      fi
>>  done
>
> (you want a pull request?)
>
> 2) The other thing is that the build tripping over UTF-8 characters.
> There are a couple in the source comments (those are trivial), but the
> more important one was that I got an 'invalid byte sequence error' when
> inserting names; I assume it's Robert?? Setting the environment variable
> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
>
> Thanks,
>
> Joe
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

joe carlson


On Oct 23, 2018, at 2:26 AM, Julie Sullivan <[hidden email]> wrote:

1. Whoops, yes, thanks for that. I'll fix. We only use testmine for travis so I didn't catch that -- but that makes sense to use testmine on not-localhost.

2. Thanks for the fix! Not sure why you are seeing this error and we aren't though.


Yeah, I’ve run into this issue before. For the InterProScan java code, there are some characters in the comments that trip up javac. I’ve wondered why this was a problem for me and not for them; as far as I know I’m using a out-of-the-box version of java.

What I read from the docs is that the “platform default” character encoding is used. What do you see for the output of ‘locale’? I have LANG="en_US.UTF-8” but also 'LC_CTYPE=“C”. I don’t know which setting javac picks up; but this might be the issue.

Joe


On 20/10/18 00:54, Joe Carlson wrote:
Hello,
I'm (finally) trying to build InterMine 2.0. But a couple of things have popped up when I tried your testmine demo:
1) The testmine demo does not appear to use anything other than localhost for the db server. There are a couple edits needed in setup.sh and testmodel.properties:
testmine> git diff dbmodel/resources/testmodel.properties
diff --git a/testmine/dbmodel/resources/testmodel.properties b/testmine/dbmodel/resources/testmodel.properties
index e3ae168..488bcd2 100644
--- a/testmine/dbmodel/resources/testmodel.properties
+++ b/testmine/dbmodel/resources/testmodel.properties
@@ -1,9 +1,9 @@
-db.userprofile-test.datasource.serverName=localhost
+db.userprofile-test.datasource.serverName=SERVER
 db.userprofile-test.datasource.databaseName=userprofile-demo
 db.userprofile-test.datasource.user=PSQL_USER
 db.userprofile-test.datasource.password=PSQL_PWD

-db.unittest.datasource.serverName=localhost
+db.unittest.datasource.serverName=SERVER
 db.unittest.datasource.databaseName=intermine-demo
 db.unittest.datasource.user=PSQL_USER
 db.unittest.datasource.password=PSQL_PWD
testmine> git diff setup.sh
diff --git a/testmine/setup.sh b/testmine/setup.sh
index 0d13f74..41bee60 100755
--- a/testmine/setup.sh
+++ b/testmine/setup.sh
@@ -74,11 +74,11 @@ fi

 echo "------> Checking databases..."
 for db in $USERPROFILEDB $PRODDB; do
-    if psql --list | egrep -q '\s'$db'\s'; then
+    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
         echo $db exists.
     else
         echo Creating $db
-        createdb $db
+        createdb -h $SERVER -E UTF8 $db
     fi
 done
(you want a pull request?)
2) The other thing is that the build tripping over UTF-8 characters. There are a couple in the source comments (those are trivial), but the more important one was that I got an 'invalid byte sequence error' when inserting names; I assume it's Robert?? Setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
Thanks,
Joe
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev


_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

joe carlson
In reply to this post by Julie Sullivan-2
Hi Julie,

I'm back looking at the gradle build. A couple other things popped up:

1) setup.sh in biotestmine is also not coded for a non-localhost
postgres server.

2) the more puzzling thing (for me, at least) is that gradle.properties
specifies intermine plugin version 3.0+. From what I see, these have not
been uploaded to jcenter (2.1.1 seems to be the latest).

Or am I doing something wrong here?

Joe

On 10/23/2018 02:26 AM, Julie Sullivan wrote:

> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for
> travis so I didn't catch that -- but that makes sense to use testmine
> on not-localhost.
>
> 2. Thanks for the fix! Not sure why you are seeing this error and we
> aren't though.
>
> On 20/10/18 00:54, Joe Carlson wrote:
>> Hello,
>>
>> I'm (finally) trying to build InterMine 2.0. But a couple of things
>> have popped up when I tried your testmine demo:
>>
>> 1) The testmine demo does not appear to use anything other than
>> localhost for the db server. There are a couple edits needed in
>> setup.sh and testmodel.properties:
>>
>>> testmine> git diff dbmodel/resources/testmodel.properties
>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>> b/testmine/dbmodel/resources/testmodel.properties
>>> index e3ae168..488bcd2 100644
>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>> @@ -1,9 +1,9 @@
>>> -db.userprofile-test.datasource.serverName=localhost
>>> +db.userprofile-test.datasource.serverName=SERVER
>>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>>  db.userprofile-test.datasource.user=PSQL_USER
>>>  db.userprofile-test.datasource.password=PSQL_PWD
>>>
>>> -db.unittest.datasource.serverName=localhost
>>> +db.unittest.datasource.serverName=SERVER
>>>  db.unittest.datasource.databaseName=intermine-demo
>>>  db.unittest.datasource.user=PSQL_USER
>>>  db.unittest.datasource.password=PSQL_PWD
>>> testmine> git diff setup.sh
>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>> index 0d13f74..41bee60 100755
>>> --- a/testmine/setup.sh
>>> +++ b/testmine/setup.sh
>>> @@ -74,11 +74,11 @@ fi
>>>
>>>  echo "------> Checking databases..."
>>>  for db in $USERPROFILEDB $PRODDB; do
>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>          echo $db exists.
>>>      else
>>>          echo Creating $db
>>> -        createdb $db
>>> +        createdb -h $SERVER -E UTF8 $db
>>>      fi
>>>  done
>>
>> (you want a pull request?)
>>
>> 2) The other thing is that the build tripping over UTF-8 characters.
>> There are a couple in the source comments (those are trivial), but
>> the more important one was that I got an 'invalid byte sequence
>> error' when inserting names; I assume it's Robert?? Setting the
>> environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears
>> to have fixed that.
>>
>> Thanks,
>>
>> Joe
>>
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

Julie Sullivan-2
Hi Joe!

1. Yes, this is on purpose (for Travis) but you can have a local
biotestmine.properties that will be used instead of the default:

https://github.com/intermine/biotestmine/blob/master/setup.sh#L80

So the default properties file has localhost, but you can use your own
properties file if you need different settings. Is that sufficient?
Again, this script was written for travis but I can update.

2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got
overzealous. You are right, 2.1.1 is the latest release.

You are not doing anything wrong! :)
Julie

On 02/11/2018 22:25, Joe Carlson wrote:

> Hi Julie,
>
> I'm back looking at the gradle build. A couple other things popped up:
>
> 1) setup.sh in biotestmine is also not coded for a non-localhost
> postgres server.
>
> 2) the more puzzling thing (for me, at least) is that gradle.properties
> specifies intermine plugin version 3.0+. From what I see, these have not
> been uploaded to jcenter (2.1.1 seems to be the latest).
>
> Or am I doing something wrong here?
>
> Joe
>
> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for
>> travis so I didn't catch that -- but that makes sense to use testmine
>> on not-localhost.
>>
>> 2. Thanks for the fix! Not sure why you are seeing this error and we
>> aren't though.
>>
>> On 20/10/18 00:54, Joe Carlson wrote:
>>> Hello,
>>>
>>> I'm (finally) trying to build InterMine 2.0. But a couple of things
>>> have popped up when I tried your testmine demo:
>>>
>>> 1) The testmine demo does not appear to use anything other than
>>> localhost for the db server. There are a couple edits needed in
>>> setup.sh and testmodel.properties:
>>>
>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>>> b/testmine/dbmodel/resources/testmodel.properties
>>>> index e3ae168..488bcd2 100644
>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>> @@ -1,9 +1,9 @@
>>>> -db.userprofile-test.datasource.serverName=localhost
>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>  db.userprofile-test.datasource.user=PSQL_USER
>>>>  db.userprofile-test.datasource.password=PSQL_PWD
>>>>
>>>> -db.unittest.datasource.serverName=localhost
>>>> +db.unittest.datasource.serverName=SERVER
>>>>  db.unittest.datasource.databaseName=intermine-demo
>>>>  db.unittest.datasource.user=PSQL_USER
>>>>  db.unittest.datasource.password=PSQL_PWD
>>>> testmine> git diff setup.sh
>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>> index 0d13f74..41bee60 100755
>>>> --- a/testmine/setup.sh
>>>> +++ b/testmine/setup.sh
>>>> @@ -74,11 +74,11 @@ fi
>>>>
>>>>  echo "------> Checking databases..."
>>>>  for db in $USERPROFILEDB $PRODDB; do
>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>          echo $db exists.
>>>>      else
>>>>          echo Creating $db
>>>> -        createdb $db
>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>      fi
>>>>  done
>>>
>>> (you want a pull request?)
>>>
>>> 2) The other thing is that the build tripping over UTF-8 characters.
>>> There are a couple in the source comments (those are trivial), but
>>> the more important one was that I got an 'invalid byte sequence
>>> error' when inserting names; I assume it's Robert?? Setting the
>>> environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears
>>> to have fixed that.
>>>
>>> Thanks,
>>>
>>> Joe
>>>
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

joe carlson


On 11/05/2018 12:17 AM, Julie Sullivan wrote:

> Hi Joe!
>
> 1. Yes, this is on purpose (for Travis) but you can have a local
> biotestmine.properties that will be used instead of the default:
>
> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>
> So the default properties file has localhost, but you can use your own
> properties file if you need different settings. Is that sufficient?
> Again, this script was written for travis but I can update.
The build will work with your own properties file, but setup.sh will
fail. I don't know if you intend this script to be run by mortals or not.

setup.sh has commands for checking and creating the database if needed
(lines 94 and 98). These commands to not specify a postgresql host. Yes,
you could create the db's manually ahead of time, but the script will
fail if you do not have a server running locally.

(adding a -h $SERVER will fix it.)

Joe
>
> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got
> overzealous. You are right, 2.1.1 is the latest release.
>
> You are not doing anything wrong! :)

I'm still new to gradle. Just making sure I'm getting it.

> Julie
>
> On 02/11/2018 22:25, Joe Carlson wrote:
>> Hi Julie,
>>
>> I'm back looking at the gradle build. A couple other things popped up:
>>
>> 1) setup.sh in biotestmine is also not coded for a non-localhost
>> postgres server.
>>
>> 2) the more puzzling thing (for me, at least) is that
>> gradle.properties specifies intermine plugin version 3.0+. From what
>> I see, these have not been uploaded to jcenter (2.1.1 seems to be the
>> latest).
>>
>> Or am I doing something wrong here?
>>
>> Joe
>>
>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for
>>> travis so I didn't catch that -- but that makes sense to use
>>> testmine on not-localhost.
>>>
>>> 2. Thanks for the fix! Not sure why you are seeing this error and we
>>> aren't though.
>>>
>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>> Hello,
>>>>
>>>> I'm (finally) trying to build InterMine 2.0. But a couple of things
>>>> have popped up when I tried your testmine demo:
>>>>
>>>> 1) The testmine demo does not appear to use anything other than
>>>> localhost for the db server. There are a couple edits needed in
>>>> setup.sh and testmodel.properties:
>>>>
>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>>>> b/testmine/dbmodel/resources/testmodel.properties
>>>>> index e3ae168..488bcd2 100644
>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>> @@ -1,9 +1,9 @@
>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>  db.userprofile-test.datasource.user=PSQL_USER
>>>>>  db.userprofile-test.datasource.password=PSQL_PWD
>>>>>
>>>>> -db.unittest.datasource.serverName=localhost
>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>  db.unittest.datasource.databaseName=intermine-demo
>>>>>  db.unittest.datasource.user=PSQL_USER
>>>>>  db.unittest.datasource.password=PSQL_PWD
>>>>> testmine> git diff setup.sh
>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>> index 0d13f74..41bee60 100755
>>>>> --- a/testmine/setup.sh
>>>>> +++ b/testmine/setup.sh
>>>>> @@ -74,11 +74,11 @@ fi
>>>>>
>>>>>  echo "------> Checking databases..."
>>>>>  for db in $USERPROFILEDB $PRODDB; do
>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>          echo $db exists.
>>>>>      else
>>>>>          echo Creating $db
>>>>> -        createdb $db
>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>      fi
>>>>>  done
>>>>
>>>> (you want a pull request?)
>>>>
>>>> 2) The other thing is that the build tripping over UTF-8
>>>> characters. There are a couple in the source comments (those are
>>>> trivial), but the more important one was that I got an 'invalid
>>>> byte sequence error' when inserting names; I assume it's Robert??
>>>> Setting the environment variable
>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
>>>>
>>>> Thanks,
>>>>
>>>> Joe
>>>>
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

joe carlson
In reply to this post by Julie Sullivan-2

Hi Julie,

I'm still migrating to gradle. I have a question about the global additions to the data model.

You have this in the docs:

Global Additions File

Previously the data model was merged from all data sources’ additions XML file (plus the SO terms, core and genomic additons). This is no longer true. Since each source is in its own JAR now, the data model is self-contained for each data source. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source:

// [in bio/sources/build.gradle]
// uncomment to specify an extra additions file for your bio-sources
// this file will be merged with the additions file for each data source
// and included in each source JAR.
//bioSourceDBModelConfig {
//    extraAdditionsFile = "MY-MINE_additions.xml"
//}


which build.gradle file are you referring to? I have a directory phytomine/ from the make_mine script; I assumed it was in this build.gradle file (there is not a bio/sources/ directory below this.), but trying to build the db was giving the message "Could not find method bioSourceDBModelConfig()". Also tried to put it in the top level of my phytomine-bio-sources. Same story.

Thanks,

Joe

On 11/5/18 12:17 AM, Julie Sullivan wrote:
Hi Joe!

1. Yes, this is on purpose (for Travis) but you can have a local biotestmine.properties that will be used instead of the default:

https://github.com/intermine/biotestmine/blob/master/setup.sh#L80

So the default properties file has localhost, but you can use your own properties file if you need different settings. Is that sufficient? Again, this script was written for travis but I can update.

2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got overzealous. You are right, 2.1.1 is the latest release.

You are not doing anything wrong! :)
Julie

On 02/11/2018 22:25, Joe Carlson wrote:
Hi Julie,

I'm back looking at the gradle build. A couple other things popped up:

1) setup.sh in biotestmine is also not coded for a non-localhost postgres server.

2) the more puzzling thing (for me, at least) is that gradle.properties specifies intermine plugin version 3.0+. From what I see, these have not been uploaded to jcenter (2.1.1 seems to be the latest).

Or am I doing something wrong here?

Joe

On 10/23/2018 02:26 AM, Julie Sullivan wrote:
1. Whoops, yes, thanks for that. I'll fix. We only use testmine for travis so I didn't catch that -- but that makes sense to use testmine on not-localhost.

2. Thanks for the fix! Not sure why you are seeing this error and we aren't though.

On 20/10/18 00:54, Joe Carlson wrote:
Hello,

I'm (finally) trying to build InterMine 2.0. But a couple of things have popped up when I tried your testmine demo:

1) The testmine demo does not appear to use anything other than localhost for the db server. There are a couple edits needed in setup.sh and testmodel.properties:

testmine> git diff dbmodel/resources/testmodel.properties
diff --git a/testmine/dbmodel/resources/testmodel.properties b/testmine/dbmodel/resources/testmodel.properties
index e3ae168..488bcd2 100644
--- a/testmine/dbmodel/resources/testmodel.properties
+++ b/testmine/dbmodel/resources/testmodel.properties
@@ -1,9 +1,9 @@
-db.userprofile-test.datasource.serverName=localhost
+db.userprofile-test.datasource.serverName=SERVER
 db.userprofile-test.datasource.databaseName=userprofile-demo
 db.userprofile-test.datasource.user=PSQL_USER
 db.userprofile-test.datasource.password=PSQL_PWD

-db.unittest.datasource.serverName=localhost
+db.unittest.datasource.serverName=SERVER
 db.unittest.datasource.databaseName=intermine-demo
 db.unittest.datasource.user=PSQL_USER
 db.unittest.datasource.password=PSQL_PWD
testmine> git diff setup.sh
diff --git a/testmine/setup.sh b/testmine/setup.sh
index 0d13f74..41bee60 100755
--- a/testmine/setup.sh
+++ b/testmine/setup.sh
@@ -74,11 +74,11 @@ fi

 echo "------> Checking databases..."
 for db in $USERPROFILEDB $PRODDB; do
-    if psql --list | egrep -q '\s'$db'\s'; then
+    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
         echo $db exists.
     else
         echo Creating $db
-        createdb $db
+        createdb -h $SERVER -E UTF8 $db
     fi
 done

(you want a pull request?)

2) The other thing is that the build tripping over UTF-8 characters. There are a couple in the source comments (those are trivial), but the more important one was that I got an 'invalid byte sequence error' when inserting names; I assume it's Robert?? Setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.

Thanks,

Joe

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev


_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

Julie Sullivan-2
Hi Joe

The top level of your phytomine's bio/sources directory is the correct
placement for that additions file, and that parameter! However the docs
were out of date, I renamed that parameter to be:

        globalAdditionsFile

because I thought it would be less confusing! :( I've now updated the
docs, apologies.

You can look at flymine here:

https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70

  globalAdditionsFile = "flymine-global_additions.xml"

And that file (flymine-global_additions.xml) lives in the root directory
of flymine-bio-sources.

Let me know if you have any more issues!
Julie


On 13/11/2018 01:49, Joe Carlson wrote:

> Hi Julie,
>
> I'm still migrating to gradle. I have a question about the global
> additions to the data model.
>
> You have this in the docs:
>
>
>     Global Additions File
>
> Previously the data model was merged from all data sources’ additions
> XML file (plus the SO terms, core and genomic additons). This is no
> longer true. Since each source is in its own JAR now, the data model is
> self-contained for each data source. Therefore if you reference a class
> in your data parser, it must be present in the additions file.
> Alternatively, you can specify a single data model file that will be
> merged into each source:
>
> //[in bio/sources/build.gradle]
> // uncomment to specify an extra additions filefor  your bio-sources
> // this file will be merged with the additions filefor  each datasource
> // and included in eachsource JAR.
> //bioSourceDBModelConfig{
> //extraAdditionsFile  =  "MY-MINE_additions.xml"
> //}
>
>
>
> which build.gradle file are you referring to? I have a directory
> phytomine/ from the make_mine script; I assumed it was in this
> build.gradle file (there is not a bio/sources/ directory below this.),
> but trying to build the db was giving the message "Could not find method
> bioSourceDBModelConfig()". Also tried to put it in the top level of my
> phytomine-bio-sources. Same story.
>
> Thanks,
>
> Joe
>
> On 11/5/18 12:17 AM, Julie Sullivan wrote:
>> Hi Joe!
>>
>> 1. Yes, this is on purpose (for Travis) but you can have a local
>> biotestmine.properties that will be used instead of the default:
>>
>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>>
>> So the default properties file has localhost, but you can use your own
>> properties file if you need different settings. Is that sufficient?
>> Again, this script was written for travis but I can update.
>>
>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got
>> overzealous. You are right, 2.1.1 is the latest release.
>>
>> You are not doing anything wrong! :)
>> Julie
>>
>> On 02/11/2018 22:25, Joe Carlson wrote:
>>> Hi Julie,
>>>
>>> I'm back looking at the gradle build. A couple other things popped up:
>>>
>>> 1) setup.sh in biotestmine is also not coded for a non-localhost
>>> postgres server.
>>>
>>> 2) the more puzzling thing (for me, at least) is that
>>> gradle.properties specifies intermine plugin version 3.0+. From what
>>> I see, these have not been uploaded to jcenter (2.1.1 seems to be the
>>> latest).
>>>
>>> Or am I doing something wrong here?
>>>
>>> Joe
>>>
>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for
>>>> travis so I didn't catch that -- but that makes sense to use
>>>> testmine on not-localhost.
>>>>
>>>> 2. Thanks for the fix! Not sure why you are seeing this error and we
>>>> aren't though.
>>>>
>>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>>> Hello,
>>>>>
>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of things
>>>>> have popped up when I tried your testmine demo:
>>>>>
>>>>> 1) The testmine demo does not appear to use anything other than
>>>>> localhost for the db server. There are a couple edits needed in
>>>>> setup.sh and testmodel.properties:
>>>>>
>>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>>>>> b/testmine/dbmodel/resources/testmodel.properties
>>>>>> index e3ae168..488bcd2 100644
>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>>> @@ -1,9 +1,9 @@
>>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>>  db.userprofile-test.datasource.user=PSQL_USER
>>>>>>  db.userprofile-test.datasource.password=PSQL_PWD
>>>>>>
>>>>>> -db.unittest.datasource.serverName=localhost
>>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>>  db.unittest.datasource.databaseName=intermine-demo
>>>>>>  db.unittest.datasource.user=PSQL_USER
>>>>>>  db.unittest.datasource.password=PSQL_PWD
>>>>>> testmine> git diff setup.sh
>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>>> index 0d13f74..41bee60 100755
>>>>>> --- a/testmine/setup.sh
>>>>>> +++ b/testmine/setup.sh
>>>>>> @@ -74,11 +74,11 @@ fi
>>>>>>
>>>>>>  echo "------> Checking databases..."
>>>>>>  for db in $USERPROFILEDB $PRODDB; do
>>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>>          echo $db exists.
>>>>>>      else
>>>>>>          echo Creating $db
>>>>>> -        createdb $db
>>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>>      fi
>>>>>>  done
>>>>>
>>>>> (you want a pull request?)
>>>>>
>>>>> 2) The other thing is that the build tripping over UTF-8
>>>>> characters. There are a couple in the source comments (those are
>>>>> trivial), but the more important one was that I got an 'invalid
>>>>> byte sequence error' when inserting names; I assume it's Robert??
>>>>> Setting the environment variable
>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Joe
>>>>>
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> [hidden email]
>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

joe carlson
Hi Julie,

I think I see what my issues were. I was a bit confused by the docs.

One issue was that the
https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script 
say make_source is invoked as ./make_source. But this script requires
the skeletons directory below it and as far as I can tell needs to be
invokes as /path/to/intermine-scripts/make_source.

The second issue was that
https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file 
refers to bio/sources/build.gradle. I was interpreting this as a path
and kept looking for the sources folder inside bio. Perhaps this should
be labeled as bio-sources or MYMINE-bio-sources, or something.

And the final thing is that it seems that bioSourceDBModelConfig
specific needs to be within the subprojects section of build.gradle. It
I put it at top level, I get the 'Could not find method
bioSourceDBModelConfig() for arguments...' error.

Thanks,

Joe

On 11/13/18 1:06 AM, Julie Sullivan wrote:

> Hi Joe
>
> The top level of your phytomine's bio/sources directory is the correct
> placement for that additions file, and that parameter! However the
> docs were out of date, I renamed that parameter to be:
>
>     globalAdditionsFile
>
> because I thought it would be less confusing! :( I've now updated the
> docs, apologies.
>
> You can look at flymine here:
>
> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 
>
>
>      globalAdditionsFile = "flymine-global_additions.xml"
>
> And that file (flymine-global_additions.xml) lives in the root
> directory of flymine-bio-sources.
>
> Let me know if you have any more issues!
> Julie
>
>
> On 13/11/2018 01:49, Joe Carlson wrote:
>> Hi Julie,
>>
>> I'm still migrating to gradle. I have a question about the global
>> additions to the data model.
>>
>> You have this in the docs:
>>
>>
>>     Global Additions File
>>
>> Previously the data model was merged from all data sources’ additions
>> XML file (plus the SO terms, core and genomic additons). This is no
>> longer true. Since each source is in its own JAR now, the data model
>> is self-contained for each data source. Therefore if you reference a
>> class in your data parser, it must be present in the additions file.
>> Alternatively, you can specify a single data model file that will be
>> merged into each source:
>>
>> //[in bio/sources/build.gradle]
>> // uncomment to specify an extra additions filefor  your bio-sources
>> // this file will be merged with the additions filefor  each datasource
>> // and included in eachsource JAR.
>> //bioSourceDBModelConfig{
>> //extraAdditionsFile  =  "MY-MINE_additions.xml"
>> //}
>>
>>
>>
>> which build.gradle file are you referring to? I have a directory
>> phytomine/ from the make_mine script; I assumed it was in this
>> build.gradle file (there is not a bio/sources/ directory below
>> this.), but trying to build the db was giving the message "Could not
>> find method bioSourceDBModelConfig()". Also tried to put it in the
>> top level of my phytomine-bio-sources. Same story.
>>
>> Thanks,
>>
>> Joe
>>
>> On 11/5/18 12:17 AM, Julie Sullivan wrote:
>>> Hi Joe!
>>>
>>> 1. Yes, this is on purpose (for Travis) but you can have a local
>>> biotestmine.properties that will be used instead of the default:
>>>
>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>>>
>>> So the default properties file has localhost, but you can use your
>>> own properties file if you need different settings. Is that
>>> sufficient? Again, this script was written for travis but I can update.
>>>
>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got
>>> overzealous. You are right, 2.1.1 is the latest release.
>>>
>>> You are not doing anything wrong! :)
>>> Julie
>>>
>>> On 02/11/2018 22:25, Joe Carlson wrote:
>>>> Hi Julie,
>>>>
>>>> I'm back looking at the gradle build. A couple other things popped up:
>>>>
>>>> 1) setup.sh in biotestmine is also not coded for a non-localhost
>>>> postgres server.
>>>>
>>>> 2) the more puzzling thing (for me, at least) is that
>>>> gradle.properties specifies intermine plugin version 3.0+. From
>>>> what I see, these have not been uploaded to jcenter (2.1.1 seems to
>>>> be the latest).
>>>>
>>>> Or am I doing something wrong here?
>>>>
>>>> Joe
>>>>
>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine
>>>>> for travis so I didn't catch that -- but that makes sense to use
>>>>> testmine on not-localhost.
>>>>>
>>>>> 2. Thanks for the fix! Not sure why you are seeing this error and
>>>>> we aren't though.
>>>>>
>>>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of
>>>>>> things have popped up when I tried your testmine demo:
>>>>>>
>>>>>> 1) The testmine demo does not appear to use anything other than
>>>>>> localhost for the db server. There are a couple edits needed in
>>>>>> setup.sh and testmodel.properties:
>>>>>>
>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>>>>>> b/testmine/dbmodel/resources/testmodel.properties
>>>>>>> index e3ae168..488bcd2 100644
>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>>>> @@ -1,9 +1,9 @@
>>>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>>>  db.userprofile-test.datasource.user=PSQL_USER
>>>>>>>  db.userprofile-test.datasource.password=PSQL_PWD
>>>>>>>
>>>>>>> -db.unittest.datasource.serverName=localhost
>>>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>>>  db.unittest.datasource.databaseName=intermine-demo
>>>>>>>  db.unittest.datasource.user=PSQL_USER
>>>>>>>  db.unittest.datasource.password=PSQL_PWD
>>>>>>> testmine> git diff setup.sh
>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>>>> index 0d13f74..41bee60 100755
>>>>>>> --- a/testmine/setup.sh
>>>>>>> +++ b/testmine/setup.sh
>>>>>>> @@ -74,11 +74,11 @@ fi
>>>>>>>
>>>>>>>  echo "------> Checking databases..."
>>>>>>>  for db in $USERPROFILEDB $PRODDB; do
>>>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>>>          echo $db exists.
>>>>>>>      else
>>>>>>>          echo Creating $db
>>>>>>> -        createdb $db
>>>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>>>      fi
>>>>>>>  done
>>>>>>
>>>>>> (you want a pull request?)
>>>>>>
>>>>>> 2) The other thing is that the build tripping over UTF-8
>>>>>> characters. There are a couple in the source comments (those are
>>>>>> trivial), but the more important one was that I got an 'invalid
>>>>>> byte sequence error' when inserting names; I assume it's Robert??
>>>>>> Setting the environment variable
>>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Joe
>>>>>>
>>>>>> _______________________________________________
>>>>>> dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

Julie Sullivan-2
Thanks Joe! Really valuable feedback.

I've updated the docs to reflect your comments, the instructions are
clearer now.

On 13/11/2018 17:12, Joe Carlson wrote:

> Hi Julie,
>
> I think I see what my issues were. I was a bit confused by the docs.
>
> One issue was that the
> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script 
> say make_source is invoked as ./make_source. But this script requires
> the skeletons directory below it and as far as I can tell needs to be
> invokes as /path/to/intermine-scripts/make_source.
>
> The second issue was that
> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file 
> refers to bio/sources/build.gradle. I was interpreting this as a path
> and kept looking for the sources folder inside bio. Perhaps this should
> be labeled as bio-sources or MYMINE-bio-sources, or something.
>
> And the final thing is that it seems that bioSourceDBModelConfig
> specific needs to be within the subprojects section of build.gradle. It
> I put it at top level, I get the 'Could not find method
> bioSourceDBModelConfig() for arguments...' error.
>
> Thanks,
>
> Joe
>
> On 11/13/18 1:06 AM, Julie Sullivan wrote:
>> Hi Joe
>>
>> The top level of your phytomine's bio/sources directory is the correct
>> placement for that additions file, and that parameter! However the
>> docs were out of date, I renamed that parameter to be:
>>
>>     globalAdditionsFile
>>
>> because I thought it would be less confusing! :( I've now updated the
>> docs, apologies.
>>
>> You can look at flymine here:
>>
>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 
>>
>>
>>      globalAdditionsFile = "flymine-global_additions.xml"
>>
>> And that file (flymine-global_additions.xml) lives in the root
>> directory of flymine-bio-sources.
>>
>> Let me know if you have any more issues!
>> Julie
>>
>>
>> On 13/11/2018 01:49, Joe Carlson wrote:
>>> Hi Julie,
>>>
>>> I'm still migrating to gradle. I have a question about the global
>>> additions to the data model.
>>>
>>> You have this in the docs:
>>>
>>>
>>>     Global Additions File
>>>
>>> Previously the data model was merged from all data sources’ additions
>>> XML file (plus the SO terms, core and genomic additons). This is no
>>> longer true. Since each source is in its own JAR now, the data model
>>> is self-contained for each data source. Therefore if you reference a
>>> class in your data parser, it must be present in the additions file.
>>> Alternatively, you can specify a single data model file that will be
>>> merged into each source:
>>>
>>> //[in bio/sources/build.gradle]
>>> // uncomment to specify an extra additions filefor  your bio-sources
>>> // this file will be merged with the additions filefor  each datasource
>>> // and included in eachsource JAR.
>>> //bioSourceDBModelConfig{
>>> //extraAdditionsFile  =  "MY-MINE_additions.xml"
>>> //}
>>>
>>>
>>>
>>> which build.gradle file are you referring to? I have a directory
>>> phytomine/ from the make_mine script; I assumed it was in this
>>> build.gradle file (there is not a bio/sources/ directory below
>>> this.), but trying to build the db was giving the message "Could not
>>> find method bioSourceDBModelConfig()". Also tried to put it in the
>>> top level of my phytomine-bio-sources. Same story.
>>>
>>> Thanks,
>>>
>>> Joe
>>>
>>> On 11/5/18 12:17 AM, Julie Sullivan wrote:
>>>> Hi Joe!
>>>>
>>>> 1. Yes, this is on purpose (for Travis) but you can have a local
>>>> biotestmine.properties that will be used instead of the default:
>>>>
>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>>>>
>>>> So the default properties file has localhost, but you can use your
>>>> own properties file if you need different settings. Is that
>>>> sufficient? Again, this script was written for travis but I can update.
>>>>
>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got
>>>> overzealous. You are right, 2.1.1 is the latest release.
>>>>
>>>> You are not doing anything wrong! :)
>>>> Julie
>>>>
>>>> On 02/11/2018 22:25, Joe Carlson wrote:
>>>>> Hi Julie,
>>>>>
>>>>> I'm back looking at the gradle build. A couple other things popped up:
>>>>>
>>>>> 1) setup.sh in biotestmine is also not coded for a non-localhost
>>>>> postgres server.
>>>>>
>>>>> 2) the more puzzling thing (for me, at least) is that
>>>>> gradle.properties specifies intermine plugin version 3.0+. From
>>>>> what I see, these have not been uploaded to jcenter (2.1.1 seems to
>>>>> be the latest).
>>>>>
>>>>> Or am I doing something wrong here?
>>>>>
>>>>> Joe
>>>>>
>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine
>>>>>> for travis so I didn't catch that -- but that makes sense to use
>>>>>> testmine on not-localhost.
>>>>>>
>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error and
>>>>>> we aren't though.
>>>>>>
>>>>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of
>>>>>>> things have popped up when I tried your testmine demo:
>>>>>>>
>>>>>>> 1) The testmine demo does not appear to use anything other than
>>>>>>> localhost for the db server. There are a couple edits needed in
>>>>>>> setup.sh and testmodel.properties:
>>>>>>>
>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>> b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>> index e3ae168..488bcd2 100644
>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>> @@ -1,9 +1,9 @@
>>>>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>>>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>>>>  db.userprofile-test.datasource.user=PSQL_USER
>>>>>>>>  db.userprofile-test.datasource.password=PSQL_PWD
>>>>>>>>
>>>>>>>> -db.unittest.datasource.serverName=localhost
>>>>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>>>>  db.unittest.datasource.databaseName=intermine-demo
>>>>>>>>  db.unittest.datasource.user=PSQL_USER
>>>>>>>>  db.unittest.datasource.password=PSQL_PWD
>>>>>>>> testmine> git diff setup.sh
>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>>>>> index 0d13f74..41bee60 100755
>>>>>>>> --- a/testmine/setup.sh
>>>>>>>> +++ b/testmine/setup.sh
>>>>>>>> @@ -74,11 +74,11 @@ fi
>>>>>>>>
>>>>>>>>  echo "------> Checking databases..."
>>>>>>>>  for db in $USERPROFILEDB $PRODDB; do
>>>>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>>>>          echo $db exists.
>>>>>>>>      else
>>>>>>>>          echo Creating $db
>>>>>>>> -        createdb $db
>>>>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>>>>      fi
>>>>>>>>  done
>>>>>>>
>>>>>>> (you want a pull request?)
>>>>>>>
>>>>>>> 2) The other thing is that the build tripping over UTF-8
>>>>>>> characters. There are a couple in the source comments (those are
>>>>>>> trivial), but the more important one was that I got an 'invalid
>>>>>>> byte sequence error' when inserting names; I assume it's Robert??
>>>>>>> Setting the environment variable
>>>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Joe
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

joe carlson
Thanks for the updates.

Another thing I noticed is a problem with the docs which has reared its ugly head again.

Type ‘merging’ into the search box. The first two links I get are ‘Model Merging’ and ‘Data Model Overview’. These links work properly. But the third link ‘Database’ goes to a ‘SORRY This page does not exist yet’ error page.

The issue seems to be that links that go to /path/to/page/index do not work, but /path/to/page/anything_else are OK. And, if you take /path/to/page/index and append .html to the URL, all is good.

Joe

> On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]> wrote:
>
> Thanks Joe! Really valuable feedback.
>
> I've updated the docs to reflect your comments, the instructions are clearer now.
>
> On 13/11/2018 17:12, Joe Carlson wrote:
>> Hi Julie,
>> I think I see what my issues were. I was a bit confused by the docs.
>> One issue was that the https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script say make_source is invoked as ./make_source. But this script requires the skeletons directory below it and as far as I can tell needs to be invokes as /path/to/intermine-scripts/make_source.
>> The second issue was that https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file refers to bio/sources/build.gradle. I was interpreting this as a path and kept looking for the sources folder inside bio. Perhaps this should be labeled as bio-sources or MYMINE-bio-sources, or something.
>> And the final thing is that it seems that bioSourceDBModelConfig specific needs to be within the subprojects section of build.gradle. It I put it at top level, I get the 'Could not find method bioSourceDBModelConfig() for arguments...' error.
>> Thanks,
>> Joe
>> On 11/13/18 1:06 AM, Julie Sullivan wrote:
>>> Hi Joe
>>>
>>> The top level of your phytomine's bio/sources directory is the correct placement for that additions file, and that parameter! However the docs were out of date, I renamed that parameter to be:
>>>
>>>     globalAdditionsFile
>>>
>>> because I thought it would be less confusing! :( I've now updated the docs, apologies.
>>>
>>> You can look at flymine here:
>>>
>>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 
>>>
>>>      globalAdditionsFile = "flymine-global_additions.xml"
>>>
>>> And that file (flymine-global_additions.xml) lives in the root directory of flymine-bio-sources.
>>>
>>> Let me know if you have any more issues!
>>> Julie
>>>
>>>
>>> On 13/11/2018 01:49, Joe Carlson wrote:
>>>> Hi Julie,
>>>>
>>>> I'm still migrating to gradle. I have a question about the global additions to the data model.
>>>>
>>>> You have this in the docs:
>>>>
>>>>
>>>>     Global Additions File
>>>>
>>>> Previously the data model was merged from all data sources’ additions XML file (plus the SO terms, core and genomic additons). This is no longer true. Since each source is in its own JAR now, the data model is self-contained for each data source. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source:
>>>>
>>>> //[in bio/sources/build.gradle]
>>>> // uncomment to specify an extra additions filefor  your bio-sources
>>>> // this file will be merged with the additions filefor  each datasource
>>>> // and included in eachsource JAR.
>>>> //bioSourceDBModelConfig{
>>>> //extraAdditionsFile  =  "MY-MINE_additions.xml"
>>>> //}
>>>>
>>>>
>>>>
>>>> which build.gradle file are you referring to? I have a directory phytomine/ from the make_mine script; I assumed it was in this build.gradle file (there is not a bio/sources/ directory below this.), but trying to build the db was giving the message "Could not find method bioSourceDBModelConfig()". Also tried to put it in the top level of my phytomine-bio-sources. Same story.
>>>>
>>>> Thanks,
>>>>
>>>> Joe
>>>>
>>>> On 11/5/18 12:17 AM, Julie Sullivan wrote:
>>>>> Hi Joe!
>>>>>
>>>>> 1. Yes, this is on purpose (for Travis) but you can have a local biotestmine.properties that will be used instead of the default:
>>>>>
>>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>>>>>
>>>>> So the default properties file has localhost, but you can use your own properties file if you need different settings. Is that sufficient? Again, this script was written for travis but I can update.
>>>>>
>>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got overzealous. You are right, 2.1.1 is the latest release.
>>>>>
>>>>> You are not doing anything wrong! :)
>>>>> Julie
>>>>>
>>>>> On 02/11/2018 22:25, Joe Carlson wrote:
>>>>>> Hi Julie,
>>>>>>
>>>>>> I'm back looking at the gradle build. A couple other things popped up:
>>>>>>
>>>>>> 1) setup.sh in biotestmine is also not coded for a non-localhost postgres server.
>>>>>>
>>>>>> 2) the more puzzling thing (for me, at least) is that gradle.properties specifies intermine plugin version 3.0+. From what I see, these have not been uploaded to jcenter (2.1.1 seems to be the latest).
>>>>>>
>>>>>> Or am I doing something wrong here?
>>>>>>
>>>>>> Joe
>>>>>>
>>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for travis so I didn't catch that -- but that makes sense to use testmine on not-localhost.
>>>>>>>
>>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error and we aren't though.
>>>>>>>
>>>>>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of things have popped up when I tried your testmine demo:
>>>>>>>>
>>>>>>>> 1) The testmine demo does not appear to use anything other than localhost for the db server. There are a couple edits needed in setup.sh and testmodel.properties:
>>>>>>>>
>>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>> index e3ae168..488bcd2 100644
>>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>> @@ -1,9 +1,9 @@
>>>>>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>>>>>  db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>>>>>  db.userprofile-test.datasource.user=PSQL_USER
>>>>>>>>>  db.userprofile-test.datasource.password=PSQL_PWD
>>>>>>>>>
>>>>>>>>> -db.unittest.datasource.serverName=localhost
>>>>>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>>>>>  db.unittest.datasource.databaseName=intermine-demo
>>>>>>>>>  db.unittest.datasource.user=PSQL_USER
>>>>>>>>>  db.unittest.datasource.password=PSQL_PWD
>>>>>>>>> testmine> git diff setup.sh
>>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>>>>>> index 0d13f74..41bee60 100755
>>>>>>>>> --- a/testmine/setup.sh
>>>>>>>>> +++ b/testmine/setup.sh
>>>>>>>>> @@ -74,11 +74,11 @@ fi
>>>>>>>>>
>>>>>>>>>  echo "------> Checking databases..."
>>>>>>>>>  for db in $USERPROFILEDB $PRODDB; do
>>>>>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>>>>>          echo $db exists.
>>>>>>>>>      else
>>>>>>>>>          echo Creating $db
>>>>>>>>> -        createdb $db
>>>>>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>>>>>      fi
>>>>>>>>>  done
>>>>>>>>
>>>>>>>> (you want a pull request?)
>>>>>>>>
>>>>>>>> 2) The other thing is that the build tripping over UTF-8 characters. There are a couple in the source comments (those are trivial), but the more important one was that I got an 'invalid byte sequence error' when inserting names; I assume it's Robert?? Setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Joe
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

Julie Sullivan-2
Yes! I have seen this too, but can't really figure out what's going on:

https://github.com/intermine/intermine/issues/1497

I asked the ReadTheDocs folks and they said it might be the stylesheet
we're using as they couldn't replicate, so I'll try updating that next.


On 14/11/2018 18:33, Joe Carlson wrote:

> Thanks for the updates.
>
> Another thing I noticed is a problem with the docs which has reared its ugly head again.
>
> Type ‘merging’ into the search box. The first two links I get are ‘Model Merging’ and ‘Data Model Overview’. These links work properly. But the third link ‘Database’ goes to a ‘SORRY This page does not exist yet’ error page.
>
> The issue seems to be that links that go to /path/to/page/index do not work, but /path/to/page/anything_else are OK. And, if you take /path/to/page/index and append .html to the URL, all is good.
>
> Joe
>
>> On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]> wrote:
>>
>> Thanks Joe! Really valuable feedback.
>>
>> I've updated the docs to reflect your comments, the instructions are clearer now.
>>
>> On 13/11/2018 17:12, Joe Carlson wrote:
>>> Hi Julie,
>>> I think I see what my issues were. I was a bit confused by the docs.
>>> One issue was that the https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script say make_source is invoked as ./make_source. But this script requires the skeletons directory below it and as far as I can tell needs to be invokes as /path/to/intermine-scripts/make_source.
>>> The second issue was that https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file refers to bio/sources/build.gradle. I was interpreting this as a path and kept looking for the sources folder inside bio. Perhaps this should be labeled as bio-sources or MYMINE-bio-sources, or something.
>>> And the final thing is that it seems that bioSourceDBModelConfig specific needs to be within the subprojects section of build.gradle. It I put it at top level, I get the 'Could not find method bioSourceDBModelConfig() for arguments...' error.
>>> Thanks,
>>> Joe
>>> On 11/13/18 1:06 AM, Julie Sullivan wrote:
>>>> Hi Joe
>>>>
>>>> The top level of your phytomine's bio/sources directory is the correct placement for that additions file, and that parameter! However the docs were out of date, I renamed that parameter to be:
>>>>
>>>>      globalAdditionsFile
>>>>
>>>> because I thought it would be less confusing! :( I've now updated the docs, apologies.
>>>>
>>>> You can look at flymine here:
>>>>
>>>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70
>>>>
>>>>       globalAdditionsFile = "flymine-global_additions.xml"
>>>>
>>>> And that file (flymine-global_additions.xml) lives in the root directory of flymine-bio-sources.
>>>>
>>>> Let me know if you have any more issues!
>>>> Julie
>>>>
>>>>
>>>> On 13/11/2018 01:49, Joe Carlson wrote:
>>>>> Hi Julie,
>>>>>
>>>>> I'm still migrating to gradle. I have a question about the global additions to the data model.
>>>>>
>>>>> You have this in the docs:
>>>>>
>>>>>
>>>>>      Global Additions File
>>>>>
>>>>> Previously the data model was merged from all data sources’ additions XML file (plus the SO terms, core and genomic additons). This is no longer true. Since each source is in its own JAR now, the data model is self-contained for each data source. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source:
>>>>>
>>>>> //[in bio/sources/build.gradle]
>>>>> // uncomment to specify an extra additions filefor  your bio-sources
>>>>> // this file will be merged with the additions filefor  each datasource
>>>>> // and included in eachsource JAR.
>>>>> //bioSourceDBModelConfig{
>>>>> //extraAdditionsFile  =  "MY-MINE_additions.xml"
>>>>> //}
>>>>>
>>>>>
>>>>>
>>>>> which build.gradle file are you referring to? I have a directory phytomine/ from the make_mine script; I assumed it was in this build.gradle file (there is not a bio/sources/ directory below this.), but trying to build the db was giving the message "Could not find method bioSourceDBModelConfig()". Also tried to put it in the top level of my phytomine-bio-sources. Same story.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Joe
>>>>>
>>>>> On 11/5/18 12:17 AM, Julie Sullivan wrote:
>>>>>> Hi Joe!
>>>>>>
>>>>>> 1. Yes, this is on purpose (for Travis) but you can have a local biotestmine.properties that will be used instead of the default:
>>>>>>
>>>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>>>>>>
>>>>>> So the default properties file has localhost, but you can use your own properties file if you need different settings. Is that sufficient? Again, this script was written for travis but I can update.
>>>>>>
>>>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got overzealous. You are right, 2.1.1 is the latest release.
>>>>>>
>>>>>> You are not doing anything wrong! :)
>>>>>> Julie
>>>>>>
>>>>>> On 02/11/2018 22:25, Joe Carlson wrote:
>>>>>>> Hi Julie,
>>>>>>>
>>>>>>> I'm back looking at the gradle build. A couple other things popped up:
>>>>>>>
>>>>>>> 1) setup.sh in biotestmine is also not coded for a non-localhost postgres server.
>>>>>>>
>>>>>>> 2) the more puzzling thing (for me, at least) is that gradle.properties specifies intermine plugin version 3.0+. From what I see, these have not been uploaded to jcenter (2.1.1 seems to be the latest).
>>>>>>>
>>>>>>> Or am I doing something wrong here?
>>>>>>>
>>>>>>> Joe
>>>>>>>
>>>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use testmine for travis so I didn't catch that -- but that makes sense to use testmine on not-localhost.
>>>>>>>>
>>>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error and we aren't though.
>>>>>>>>
>>>>>>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of things have popped up when I tried your testmine demo:
>>>>>>>>>
>>>>>>>>> 1) The testmine demo does not appear to use anything other than localhost for the db server. There are a couple edits needed in setup.sh and testmodel.properties:
>>>>>>>>>
>>>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>> index e3ae168..488bcd2 100644
>>>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>> @@ -1,9 +1,9 @@
>>>>>>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>>>>>>   db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>>>>>>   db.userprofile-test.datasource.user=PSQL_USER
>>>>>>>>>>   db.userprofile-test.datasource.password=PSQL_PWD
>>>>>>>>>>
>>>>>>>>>> -db.unittest.datasource.serverName=localhost
>>>>>>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>>>>>>   db.unittest.datasource.databaseName=intermine-demo
>>>>>>>>>>   db.unittest.datasource.user=PSQL_USER
>>>>>>>>>>   db.unittest.datasource.password=PSQL_PWD
>>>>>>>>>> testmine> git diff setup.sh
>>>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>>>>>>> index 0d13f74..41bee60 100755
>>>>>>>>>> --- a/testmine/setup.sh
>>>>>>>>>> +++ b/testmine/setup.sh
>>>>>>>>>> @@ -74,11 +74,11 @@ fi
>>>>>>>>>>
>>>>>>>>>>   echo "------> Checking databases..."
>>>>>>>>>>   for db in $USERPROFILEDB $PRODDB; do
>>>>>>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>>>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>>>>>>           echo $db exists.
>>>>>>>>>>       else
>>>>>>>>>>           echo Creating $db
>>>>>>>>>> -        createdb $db
>>>>>>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>>>>>>       fi
>>>>>>>>>>   done
>>>>>>>>>
>>>>>>>>> (you want a pull request?)
>>>>>>>>>
>>>>>>>>> 2) The other thing is that the build tripping over UTF-8 characters. There are a couple in the source comments (those are trivial), but the more important one was that I got an 'invalid byte sequence error' when inserting names; I assume it's Robert?? Setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed that.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Joe
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

joe carlson
yeah. that's weird. I remember seeing this in the past, but thought it
had been fixed.

It looks like the non-index pages are generating responding with a 301
'moved permanently' response and appending a / to the URL

> > curl -v --head
> "https://intermine.readthedocs.io/en/latest/database/data-sources/library/so"
> *   Trying 137.116.78.48...
> <snip>
> > HEAD /en/latest/database/data-sources/library/so HTTP/1.1
> > Host: intermine.readthedocs.io
> > User-Agent: curl/7.60.0
> > Accept: */*
> >
> < HTTP/1.1 301 Moved Permanently
> HTTP/1.1 301 Moved Permanently
> < Content-Length: 194
> Content-Length: 194
> < Content-Type: text/html
> Content-Type: text/html
> < Location:
> http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/
> Location:
> http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/
> < Server: nginx/1.14.0 (Ubuntu)
> Server: nginx/1.14.0 (Ubuntu)
> < X-Subdomain-TryFiles: True
> X-Subdomain-TryFiles: True
> < X-Served: Nginx
> X-Served: Nginx
> < X-Deity: web03
> X-Deity: web03
> < Date: Thu, 15 Nov 2018 16:44:46 GMT
> Date: Thu, 15 Nov 2018 16:44:46 GMT
>
> <
> * Connection #0 to host intermine.readthedocs.io left intact

But URLs that point to /index respond with a 404 without any redirect.
Are you running nginx as a proxy server or is it delivering content? It
looks like the rewrite rules for index URL's needs a little tweaking.


Joe

>> Thanks for the updates.
>>
>> Another thing I noticed is a problem with the docs which has reared
>> its ugly head again.
>>
>> Type ‘merging’ into the search box. The first two links I get are
>> ‘Model Merging’ and ‘Data Model Overview’. These links work properly.
>> But the third link ‘Database’ goes to a ‘SORRY This page does not
>> exist yet’ error page.
>>
>> The issue seems to be that links that go to /path/to/page/index do
>> not work, but /path/to/page/anything_else are OK. And, if you take
>> /path/to/page/index and append .html to the URL, all is good.
>>
>> Joe
>>
>>> On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]>
>>> wrote:
>>>
>>> Thanks Joe! Really valuable feedback.
>>>
>>> I've updated the docs to reflect your comments, the instructions are
>>> clearer now.
>>>
>>> On 13/11/2018 17:12, Joe Carlson wrote:
>>>> Hi Julie,
>>>> I think I see what my issues were. I was a bit confused by the docs.
>>>> One issue was that the
>>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script 
>>>> say make_source is invoked as ./make_source. But this script
>>>> requires the skeletons directory below it and as far as I can tell
>>>> needs to be invokes as /path/to/intermine-scripts/make_source.
>>>> The second issue was that
>>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file 
>>>> refers to bio/sources/build.gradle. I was interpreting this as a
>>>> path and kept looking for the sources folder inside bio. Perhaps
>>>> this should be labeled as bio-sources or MYMINE-bio-sources, or
>>>> something.
>>>> And the final thing is that it seems that bioSourceDBModelConfig
>>>> specific needs to be within the subprojects section of
>>>> build.gradle. It I put it at top level, I get the 'Could not find
>>>> method bioSourceDBModelConfig() for arguments...' error.
>>>> Thanks,
>>>> Joe
>>>> On 11/13/18 1:06 AM, Julie Sullivan wrote:
>>>>> Hi Joe
>>>>>
>>>>> The top level of your phytomine's bio/sources directory is the
>>>>> correct placement for that additions file, and that parameter!
>>>>> However the docs were out of date, I renamed that parameter to be:
>>>>>
>>>>>      globalAdditionsFile
>>>>>
>>>>> because I thought it would be less confusing! :( I've now updated
>>>>> the docs, apologies.
>>>>>
>>>>> You can look at flymine here:
>>>>>
>>>>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 
>>>>>
>>>>>
>>>>>       globalAdditionsFile = "flymine-global_additions.xml"
>>>>>
>>>>> And that file (flymine-global_additions.xml) lives in the root
>>>>> directory of flymine-bio-sources.
>>>>>
>>>>> Let me know if you have any more issues!
>>>>> Julie
>>>>>
>>>>>
>>>>> On 13/11/2018 01:49, Joe Carlson wrote:
>>>>>> Hi Julie,
>>>>>>
>>>>>> I'm still migrating to gradle. I have a question about the global
>>>>>> additions to the data model.
>>>>>>
>>>>>> You have this in the docs:
>>>>>>
>>>>>>
>>>>>>      Global Additions File
>>>>>>
>>>>>> Previously the data model was merged from all data sources’
>>>>>> additions XML file (plus the SO terms, core and genomic
>>>>>> additons). This is no longer true. Since each source is in its
>>>>>> own JAR now, the data model is self-contained for each data
>>>>>> source. Therefore if you reference a class in your data parser,
>>>>>> it must be present in the additions file. Alternatively, you can
>>>>>> specify a single data model file that will be merged into each
>>>>>> source:
>>>>>>
>>>>>> //[in bio/sources/build.gradle]
>>>>>> // uncomment to specify an extra additions filefor  your bio-sources
>>>>>> // this file will be merged with the additions filefor each
>>>>>> datasource
>>>>>> // and included in eachsource JAR.
>>>>>> //bioSourceDBModelConfig{
>>>>>> //extraAdditionsFile  =  "MY-MINE_additions.xml"
>>>>>> //}
>>>>>>
>>>>>>
>>>>>>
>>>>>> which build.gradle file are you referring to? I have a directory
>>>>>> phytomine/ from the make_mine script; I assumed it was in this
>>>>>> build.gradle file (there is not a bio/sources/ directory below
>>>>>> this.), but trying to build the db was giving the message "Could
>>>>>> not find method bioSourceDBModelConfig()". Also tried to put it
>>>>>> in the top level of my phytomine-bio-sources. Same story.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Joe
>>>>>>
>>>>>> On 11/5/18 12:17 AM, Julie Sullivan wrote:
>>>>>>> Hi Joe!
>>>>>>>
>>>>>>> 1. Yes, this is on purpose (for Travis) but you can have a local
>>>>>>> biotestmine.properties that will be used instead of the default:
>>>>>>>
>>>>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>>>>>>>
>>>>>>> So the default properties file has localhost, but you can use
>>>>>>> your own properties file if you need different settings. Is that
>>>>>>> sufficient? Again, this script was written for travis but I can
>>>>>>> update.
>>>>>>>
>>>>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got
>>>>>>> overzealous. You are right, 2.1.1 is the latest release.
>>>>>>>
>>>>>>> You are not doing anything wrong! :)
>>>>>>> Julie
>>>>>>>
>>>>>>> On 02/11/2018 22:25, Joe Carlson wrote:
>>>>>>>> Hi Julie,
>>>>>>>>
>>>>>>>> I'm back looking at the gradle build. A couple other things
>>>>>>>> popped up:
>>>>>>>>
>>>>>>>> 1) setup.sh in biotestmine is also not coded for a
>>>>>>>> non-localhost postgres server.
>>>>>>>>
>>>>>>>> 2) the more puzzling thing (for me, at least) is that
>>>>>>>> gradle.properties specifies intermine plugin version 3.0+. From
>>>>>>>> what I see, these have not been uploaded to jcenter (2.1.1
>>>>>>>> seems to be the latest).
>>>>>>>>
>>>>>>>> Or am I doing something wrong here?
>>>>>>>>
>>>>>>>> Joe
>>>>>>>>
>>>>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>>>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use
>>>>>>>>> testmine for travis so I didn't catch that -- but that makes
>>>>>>>>> sense to use testmine on not-localhost.
>>>>>>>>>
>>>>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error
>>>>>>>>> and we aren't though.
>>>>>>>>>
>>>>>>>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of
>>>>>>>>>> things have popped up when I tried your testmine demo:
>>>>>>>>>>
>>>>>>>>>> 1) The testmine demo does not appear to use anything other
>>>>>>>>>> than localhost for the db server. There are a couple edits
>>>>>>>>>> needed in setup.sh and testmodel.properties:
>>>>>>>>>>
>>>>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>> b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>> index e3ae168..488bcd2 100644
>>>>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>> @@ -1,9 +1,9 @@
>>>>>>>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>>>>>>> db.userprofile-test.datasource.user=PSQL_USER
>>>>>>>>>>> db.userprofile-test.datasource.password=PSQL_PWD
>>>>>>>>>>>
>>>>>>>>>>> -db.unittest.datasource.serverName=localhost
>>>>>>>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>>>>>>> db.unittest.datasource.databaseName=intermine-demo
>>>>>>>>>>>   db.unittest.datasource.user=PSQL_USER
>>>>>>>>>>>   db.unittest.datasource.password=PSQL_PWD
>>>>>>>>>>> testmine> git diff setup.sh
>>>>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>>>>>>>> index 0d13f74..41bee60 100755
>>>>>>>>>>> --- a/testmine/setup.sh
>>>>>>>>>>> +++ b/testmine/setup.sh
>>>>>>>>>>> @@ -74,11 +74,11 @@ fi
>>>>>>>>>>>
>>>>>>>>>>>   echo "------> Checking databases..."
>>>>>>>>>>>   for db in $USERPROFILEDB $PRODDB; do
>>>>>>>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>>>>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>>>>>>>           echo $db exists.
>>>>>>>>>>>       else
>>>>>>>>>>>           echo Creating $db
>>>>>>>>>>> -        createdb $db
>>>>>>>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>>>>>>>       fi
>>>>>>>>>>>   done
>>>>>>>>>>
>>>>>>>>>> (you want a pull request?)
>>>>>>>>>>
>>>>>>>>>> 2) The other thing is that the build tripping over UTF-8
>>>>>>>>>> characters. There are a couple in the source comments (those
>>>>>>>>>> are trivial), but the more important one was that I got an
>>>>>>>>>> 'invalid byte sequence error' when inserting names; I assume
>>>>>>>>>> it's Robert?? Setting the environment variable
>>>>>>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed
>>>>>>>>>> that.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Joe
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> dev mailing list
>>>>>>>>>> [hidden email]
>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>
>>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Starting on InterMine 2.0....

Julie Sullivan-2
That's what I thought! But they are hosting, and on that ticket the read
the docs dev commented they couldn't replicate. And blamed it on our
beautiful, innocent stylesheet.

On 15/11/2018 16:50, Joe Carlson wrote:

> yeah. that's weird. I remember seeing this in the past, but thought it
> had been fixed.
>
> It looks like the non-index pages are generating responding with a 301
> 'moved permanently' response and appending a / to the URL
>
>> > curl -v --head
>> "https://intermine.readthedocs.io/en/latest/database/data-sources/library/so"
>>
>> *   Trying 137.116.78.48...
>> <snip>
>> > HEAD /en/latest/database/data-sources/library/so HTTP/1.1
>> > Host: intermine.readthedocs.io
>> > User-Agent: curl/7.60.0
>> > Accept: */*
>> >
>> < HTTP/1.1 301 Moved Permanently
>> HTTP/1.1 301 Moved Permanently
>> < Content-Length: 194
>> Content-Length: 194
>> < Content-Type: text/html
>> Content-Type: text/html
>> < Location:
>> http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/ 
>>
>> Location:
>> http://intermine.readthedocs.io/en/latest/database/data-sources/library/so/ 
>>
>> < Server: nginx/1.14.0 (Ubuntu)
>> Server: nginx/1.14.0 (Ubuntu)
>> < X-Subdomain-TryFiles: True
>> X-Subdomain-TryFiles: True
>> < X-Served: Nginx
>> X-Served: Nginx
>> < X-Deity: web03
>> X-Deity: web03
>> < Date: Thu, 15 Nov 2018 16:44:46 GMT
>> Date: Thu, 15 Nov 2018 16:44:46 GMT
>>
>> <
>> * Connection #0 to host intermine.readthedocs.io left intact
>
> But URLs that point to /index respond with a 404 without any redirect.
> Are you running nginx as a proxy server or is it delivering content? It
> looks like the rewrite rules for index URL's needs a little tweaking.
>
>
> Joe
>
>>> Thanks for the updates.
>>>
>>> Another thing I noticed is a problem with the docs which has reared
>>> its ugly head again.
>>>
>>> Type ‘merging’ into the search box. The first two links I get are
>>> ‘Model Merging’ and ‘Data Model Overview’. These links work properly.
>>> But the third link ‘Database’ goes to a ‘SORRY This page does not
>>> exist yet’ error page.
>>>
>>> The issue seems to be that links that go to /path/to/page/index do
>>> not work, but /path/to/page/anything_else are OK. And, if you take
>>> /path/to/page/index and append .html to the URL, all is good.
>>>
>>> Joe
>>>
>>>> On Nov 14, 2018, at 2:24 AM, Julie Sullivan <[hidden email]>
>>>> wrote:
>>>>
>>>> Thanks Joe! Really valuable feedback.
>>>>
>>>> I've updated the docs to reflect your comments, the instructions are
>>>> clearer now.
>>>>
>>>> On 13/11/2018 17:12, Joe Carlson wrote:
>>>>> Hi Julie,
>>>>> I think I see what my issues were. I was a bit confused by the docs.
>>>>> One issue was that the
>>>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#run-make-source-script 
>>>>> say make_source is invoked as ./make_source. But this script
>>>>> requires the skeletons directory below it and as far as I can tell
>>>>> needs to be invokes as /path/to/intermine-scripts/make_source.
>>>>> The second issue was that
>>>>> https://intermine.readthedocs.io/en/latest/database/data-sources/custom/#global-additions-file 
>>>>> refers to bio/sources/build.gradle. I was interpreting this as a
>>>>> path and kept looking for the sources folder inside bio. Perhaps
>>>>> this should be labeled as bio-sources or MYMINE-bio-sources, or
>>>>> something.
>>>>> And the final thing is that it seems that bioSourceDBModelConfig
>>>>> specific needs to be within the subprojects section of
>>>>> build.gradle. It I put it at top level, I get the 'Could not find
>>>>> method bioSourceDBModelConfig() for arguments...' error.
>>>>> Thanks,
>>>>> Joe
>>>>> On 11/13/18 1:06 AM, Julie Sullivan wrote:
>>>>>> Hi Joe
>>>>>>
>>>>>> The top level of your phytomine's bio/sources directory is the
>>>>>> correct placement for that additions file, and that parameter!
>>>>>> However the docs were out of date, I renamed that parameter to be:
>>>>>>
>>>>>>      globalAdditionsFile
>>>>>>
>>>>>> because I thought it would be less confusing! :( I've now updated
>>>>>> the docs, apologies.
>>>>>>
>>>>>> You can look at flymine here:
>>>>>>
>>>>>> https://github.com/intermine/flymine-bio-sources/blob/master/build.gradle#L70 
>>>>>>
>>>>>>
>>>>>>       globalAdditionsFile = "flymine-global_additions.xml"
>>>>>>
>>>>>> And that file (flymine-global_additions.xml) lives in the root
>>>>>> directory of flymine-bio-sources.
>>>>>>
>>>>>> Let me know if you have any more issues!
>>>>>> Julie
>>>>>>
>>>>>>
>>>>>> On 13/11/2018 01:49, Joe Carlson wrote:
>>>>>>> Hi Julie,
>>>>>>>
>>>>>>> I'm still migrating to gradle. I have a question about the global
>>>>>>> additions to the data model.
>>>>>>>
>>>>>>> You have this in the docs:
>>>>>>>
>>>>>>>
>>>>>>>      Global Additions File
>>>>>>>
>>>>>>> Previously the data model was merged from all data sources’
>>>>>>> additions XML file (plus the SO terms, core and genomic
>>>>>>> additons). This is no longer true. Since each source is in its
>>>>>>> own JAR now, the data model is self-contained for each data
>>>>>>> source. Therefore if you reference a class in your data parser,
>>>>>>> it must be present in the additions file. Alternatively, you can
>>>>>>> specify a single data model file that will be merged into each
>>>>>>> source:
>>>>>>>
>>>>>>> //[in bio/sources/build.gradle]
>>>>>>> // uncomment to specify an extra additions filefor  your bio-sources
>>>>>>> // this file will be merged with the additions filefor each
>>>>>>> datasource
>>>>>>> // and included in eachsource JAR.
>>>>>>> //bioSourceDBModelConfig{
>>>>>>> //extraAdditionsFile  =  "MY-MINE_additions.xml"
>>>>>>> //}
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> which build.gradle file are you referring to? I have a directory
>>>>>>> phytomine/ from the make_mine script; I assumed it was in this
>>>>>>> build.gradle file (there is not a bio/sources/ directory below
>>>>>>> this.), but trying to build the db was giving the message "Could
>>>>>>> not find method bioSourceDBModelConfig()". Also tried to put it
>>>>>>> in the top level of my phytomine-bio-sources. Same story.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Joe
>>>>>>>
>>>>>>> On 11/5/18 12:17 AM, Julie Sullivan wrote:
>>>>>>>> Hi Joe!
>>>>>>>>
>>>>>>>> 1. Yes, this is on purpose (for Travis) but you can have a local
>>>>>>>> biotestmine.properties that will be used instead of the default:
>>>>>>>>
>>>>>>>> https://github.com/intermine/biotestmine/blob/master/setup.sh#L80
>>>>>>>>
>>>>>>>> So the default properties file has localhost, but you can use
>>>>>>>> your own properties file if you need different settings. Is that
>>>>>>>> sufficient? Again, this script was written for travis but I can
>>>>>>>> update.
>>>>>>>>
>>>>>>>> 2. Whoops! Yes, sorry, reverted now. Just testing 3.0 and got
>>>>>>>> overzealous. You are right, 2.1.1 is the latest release.
>>>>>>>>
>>>>>>>> You are not doing anything wrong! :)
>>>>>>>> Julie
>>>>>>>>
>>>>>>>> On 02/11/2018 22:25, Joe Carlson wrote:
>>>>>>>>> Hi Julie,
>>>>>>>>>
>>>>>>>>> I'm back looking at the gradle build. A couple other things
>>>>>>>>> popped up:
>>>>>>>>>
>>>>>>>>> 1) setup.sh in biotestmine is also not coded for a
>>>>>>>>> non-localhost postgres server.
>>>>>>>>>
>>>>>>>>> 2) the more puzzling thing (for me, at least) is that
>>>>>>>>> gradle.properties specifies intermine plugin version 3.0+. From
>>>>>>>>> what I see, these have not been uploaded to jcenter (2.1.1
>>>>>>>>> seems to be the latest).
>>>>>>>>>
>>>>>>>>> Or am I doing something wrong here?
>>>>>>>>>
>>>>>>>>> Joe
>>>>>>>>>
>>>>>>>>> On 10/23/2018 02:26 AM, Julie Sullivan wrote:
>>>>>>>>>> 1. Whoops, yes, thanks for that. I'll fix. We only use
>>>>>>>>>> testmine for travis so I didn't catch that -- but that makes
>>>>>>>>>> sense to use testmine on not-localhost.
>>>>>>>>>>
>>>>>>>>>> 2. Thanks for the fix! Not sure why you are seeing this error
>>>>>>>>>> and we aren't though.
>>>>>>>>>>
>>>>>>>>>> On 20/10/18 00:54, Joe Carlson wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I'm (finally) trying to build InterMine 2.0. But a couple of
>>>>>>>>>>> things have popped up when I tried your testmine demo:
>>>>>>>>>>>
>>>>>>>>>>> 1) The testmine demo does not appear to use anything other
>>>>>>>>>>> than localhost for the db server. There are a couple edits
>>>>>>>>>>> needed in setup.sh and testmodel.properties:
>>>>>>>>>>>
>>>>>>>>>>>> testmine> git diff dbmodel/resources/testmodel.properties
>>>>>>>>>>>> diff --git a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>>> b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>>> index e3ae168..488bcd2 100644
>>>>>>>>>>>> --- a/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>>> +++ b/testmine/dbmodel/resources/testmodel.properties
>>>>>>>>>>>> @@ -1,9 +1,9 @@
>>>>>>>>>>>> -db.userprofile-test.datasource.serverName=localhost
>>>>>>>>>>>> +db.userprofile-test.datasource.serverName=SERVER
>>>>>>>>>>>> db.userprofile-test.datasource.databaseName=userprofile-demo
>>>>>>>>>>>> db.userprofile-test.datasource.user=PSQL_USER
>>>>>>>>>>>> db.userprofile-test.datasource.password=PSQL_PWD
>>>>>>>>>>>>
>>>>>>>>>>>> -db.unittest.datasource.serverName=localhost
>>>>>>>>>>>> +db.unittest.datasource.serverName=SERVER
>>>>>>>>>>>> db.unittest.datasource.databaseName=intermine-demo
>>>>>>>>>>>>   db.unittest.datasource.user=PSQL_USER
>>>>>>>>>>>>   db.unittest.datasource.password=PSQL_PWD
>>>>>>>>>>>> testmine> git diff setup.sh
>>>>>>>>>>>> diff --git a/testmine/setup.sh b/testmine/setup.sh
>>>>>>>>>>>> index 0d13f74..41bee60 100755
>>>>>>>>>>>> --- a/testmine/setup.sh
>>>>>>>>>>>> +++ b/testmine/setup.sh
>>>>>>>>>>>> @@ -74,11 +74,11 @@ fi
>>>>>>>>>>>>
>>>>>>>>>>>>   echo "------> Checking databases..."
>>>>>>>>>>>>   for db in $USERPROFILEDB $PRODDB; do
>>>>>>>>>>>> -    if psql --list | egrep -q '\s'$db'\s'; then
>>>>>>>>>>>> +    if psql -h $SERVER --list | egrep -q '\s'$db'\s'; then
>>>>>>>>>>>>           echo $db exists.
>>>>>>>>>>>>       else
>>>>>>>>>>>>           echo Creating $db
>>>>>>>>>>>> -        createdb $db
>>>>>>>>>>>> +        createdb -h $SERVER -E UTF8 $db
>>>>>>>>>>>>       fi
>>>>>>>>>>>>   done
>>>>>>>>>>>
>>>>>>>>>>> (you want a pull request?)
>>>>>>>>>>>
>>>>>>>>>>> 2) The other thing is that the build tripping over UTF-8
>>>>>>>>>>> characters. There are a couple in the source comments (those
>>>>>>>>>>> are trivial), but the more important one was that I got an
>>>>>>>>>>> 'invalid byte sequence error' when inserting names; I assume
>>>>>>>>>>> it's Robert?? Setting the environment variable
>>>>>>>>>>> JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 appears to have fixed
>>>>>>>>>>> that.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Joe
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> dev mailing list
>>>>>>>>>>> [hidden email]
>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>
>>>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev