Quantcast

silly ant question

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

silly ant question

joe carlson
Hello,

I suppose this is part of my formiphobia, but I'm stuck on a question
about ant, particularly how the classpath is manipulated.

I'm trying to add a task to the loading process. It's just silly thing
that posts a message to slack to let us know that a loading step has
completed. During this time I want to access intermine.properties. I
need to use the version that is constructed from
~/.intermine/<mine>.properties since it has a secret token in it. The
composite properties file intermine.properties in build/main

I have an element within integrate

    <slack-notify source="${source}" properties="intermine.properties" />

and the corresponding code in SlackNotify.java

>  public void setProperties(String propertiesFile) {
>     projectProperties = new Properties();
>     try {
>       ClassLoader cl = SlackNotify.class.getClassLoader();
>       InputStream is = cl.getResourceAsStream(propertiesFile);
>        if (is == null) {
>           throw new BuildException("Could not find file " +
> propertiesFile);
>       }
>       projectProperties.load(is);
>       is.close();
>   } catch (IOException e) {
>       throw new BuildException("Failed to load :" + propertiesFile, e);
>   }
> }

This does not run since ./build/main is not in the classpath. If I look
at the variable cl in the debugger, I see that the only element in the
classpath is im-ant-tasks.jar. I can manually add the directory to the
classpath with

>   AntClassLoader cl = (AntClassLoader) SlackNotify.class.getClassLoader();
> cl.addPathElement(getProject().getBaseDir().toString()+"/build/main");
That's just wrong. It works, but I don't like this solution.

But, here's the thing, when I look at loadProperties in
org.intermine.metadata.Util, it has essentially the same code

>    protected static Properties loadProperties(String filename) {
>         Properties props = new NonOverrideableProperties();
>         try {
>             ClassLoader loader = Util.class.getClassLoader();
>             InputStream is = loader.getResourceAsStream(filename);
>             if (is == null) {
>                 LOG.error("Could not find file " + filename + " from "
> + loader);
>                 return null;
>             }
>             props.load(is);
>             is.close();
>         } catch (IOException e) {
>             throw new RuntimeException("Failed to load :" + filename, e);
>         }
>         return props;
>     }

This code is executed as part of the integration step as well. When I
look at the variable loader, it contains the element of the classpath
that I'm hoping to access. It certainly makes sense that Integrate.java
(or something it calls) has access to the properties files since it uses
that in establishing the db connections. As far as I can tell, when the
code for execute() in Integrate.java is called, the classpath is still
only im-ant-tasks.jar. But later the classpath when Util.class is called
includes the directories I'm looking for.

somehow, somewhere in task.xml or integrate.xml or something, there is a
mechanism for setting the classpath. And I assume some class is doing
the manipulation of the classpath. I had thought that it would involve
tossing in
>    <path id="task.class.path">
>       <pathelement location="${build.task.dir}"/>
>    </path>
somewhere. But, where? That seems to be where I'm stuck. How do I set
the classpath in the xml files so that my code can find
intermine.properties?

Can someone give me a tip on setting up the classpath properly?

Thanks

Joe
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: silly ant question

justincc
Hi Joe,

I don't think this is a silly question at all - I think InterMine's
build system is convoluted and hard to pick through.

In this case, I believe (though I could be wrong) that the
task.class.path is only used in loading tasks to ant, not in their
execution.  I believe that setting up the classpath for execution
involves invoking org.intermine.tasks.Dependencies, which is chiefly
triggered via the internal -init-deps target in library.xml.

As you can see from integrate.xml, the "integrate" target depends on
"-init-deps", which in integrate.xml then calls through to the imported
library.init-deps

The Dependencies class sets up a bunch of ant classpath properties via
its execute() method, including project.execute.path, for instance,
which is used as the classpath in several places.  I suspect
Util.loadProperties() by a task after this has been set up, though again
I could be wrong.  Invoking your task may require making sure that
-init-deps (or maybe even just <dependencies>) has been triggered at
some prior point.

Exactly how are you defining the target that triggers <slack-notify>?

On 2016-05-04 06:59, Joe Carlson wrote:

> Hello,
>
> I suppose this is part of my formiphobia, but I'm stuck on a question
> about ant, particularly how the classpath is manipulated.
>
> I'm trying to add a task to the loading process. It's just silly thing
> that posts a message to slack to let us know that a loading step has
> completed. During this time I want to access intermine.properties. I
> need to use the version that is constructed from
> ~/.intermine/<mine>.properties since it has a secret token in it. The
> composite properties file intermine.properties in build/main
>
> I have an element within integrate
>
>    <slack-notify source="${source}" properties="intermine.properties"
> />
>
> and the corresponding code in SlackNotify.java
>
>>  public void setProperties(String propertiesFile) {
>>     projectProperties = new Properties();
>>     try {
>>       ClassLoader cl = SlackNotify.class.getClassLoader();
>>       InputStream is = cl.getResourceAsStream(propertiesFile);
>>        if (is == null) {
>>           throw new BuildException("Could not find file " +
>> propertiesFile);
>>       }
>>       projectProperties.load(is);
>>       is.close();
>>   } catch (IOException e) {
>>       throw new BuildException("Failed to load :" + propertiesFile,
>> e);
>>   }
>> }
>
> This does not run since ./build/main is not in the classpath. If I
> look at the variable cl in the debugger, I see that the only element
> in the classpath is im-ant-tasks.jar. I can manually add the directory
> to the classpath with
>
>>   AntClassLoader cl = (AntClassLoader)
>> SlackNotify.class.getClassLoader();
>> cl.addPathElement(getProject().getBaseDir().toString()+"/build/main");
> That's just wrong. It works, but I don't like this solution.
>
> But, here's the thing, when I look at loadProperties in
> org.intermine.metadata.Util, it has essentially the same code
>>    protected static Properties loadProperties(String filename) {
>>         Properties props = new NonOverrideableProperties();
>>         try {
>>             ClassLoader loader = Util.class.getClassLoader();
>>             InputStream is = loader.getResourceAsStream(filename);
>>             if (is == null) {
>>                 LOG.error("Could not find file " + filename + " from "
>> + loader);
>>                 return null;
>>             }
>>             props.load(is);
>>             is.close();
>>         } catch (IOException e) {
>>             throw new RuntimeException("Failed to load :" + filename,
>> e);
>>         }
>>         return props;
>>     }
>
> This code is executed as part of the integration step as well. When I
> look at the variable loader, it contains the element of the classpath
> that I'm hoping to access. It certainly makes sense that
> Integrate.java (or something it calls) has access to the properties
> files since it uses that in establishing the db connections. As far as
> I can tell, when the code for execute() in Integrate.java is called,
> the classpath is still only im-ant-tasks.jar. But later the classpath
> when Util.class is called includes the directories I'm looking for.
>
> somehow, somewhere in task.xml or integrate.xml or something, there is
> a mechanism for setting the classpath. And I assume some class is
> doing the manipulation of the classpath. I had thought that it would
> involve tossing in
>>    <path id="task.class.path">
>>       <pathelement location="${build.task.dir}"/>
>>    </path>
> somewhere. But, where? That seems to be where I'm stuck. How do I set
> the classpath in the xml files so that my code can find
> intermine.properties?
>
> Can someone give me a tip on setting up the classpath properly?
>
> Thanks
>
> Joe
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: silly ant question

joe carlson

On May 4, 2016, at 1:07 PM, [hidden email] wrote:

Hi Joe,

I don't think this is a silly question at all - I think InterMine's build system is convoluted and hard to pick through.


thanks. It’s not just me who thought the build system was tough.

In this case, I believe (though I could be wrong) that the task.class.path is only used in loading tasks to ant, not in their execution.  I believe that setting up the classpath for execution involves invoking org.intermine.tasks.Dependencies, which is chiefly triggered via the internal -init-deps target in library.xml.

As you can see from integrate.xml, the "integrate" target depends on "-init-deps", which in integrate.xml then calls through to the imported library.init-deps

The Dependencies class sets up a bunch of ant classpath properties via its execute() method, including project.execute.path, for instance, which is used as the classpath in several places.  I suspect Util.loadProperties() by a task after this has been set up, though again I could be wrong.  Invoking your task may require making sure that -init-deps (or maybe even just <dependencies>) has been triggered at some prior point.

Exactly how are you defining the target that triggers <slack-notify>?

this is probably where my problem is. As soon as I try to determine what is a target and what is a task and what is a typedef… my eyes glaze over. And it may not help that ‘integrate’ is both a target and a task.

I thought I could get a new task into the system by adding to antlib.xml:
       <typedef name="slack-notify" classname="org.intermine.task.SlackNotify”/>

and add the task to the integrate target in integrate.xml:

 <target name="integrate" depends="-init-properties, init, -init-deps, -init-integrate">
    <integrate projectXml="../project.xml" source="${source}" action="${action}"/>
    <!-- not standard. -->
    <slack-notify source="${source}" properties="intermine.properties" />
  </target>

This passes intermine.properties into the setProperties routine. And it defines the source so that the notify messages captures that. This step does depend on -init-deps, so that part is good.

It was mostly very simple to add a task. The one last bit about the class path has me stumped.

Joe

On 2016-05-04 06:59, Joe Carlson wrote:
Hello,
I suppose this is part of my formiphobia, but I'm stuck on a question
about ant, particularly how the classpath is manipulated.
I'm trying to add a task to the loading process. It's just silly thing
that posts a message to slack to let us know that a loading step has
completed. During this time I want to access intermine.properties. I
need to use the version that is constructed from
~/.intermine/<mine>.properties since it has a secret token in it. The
composite properties file intermine.properties in build/main
I have an element within integrate
  <slack-notify source="${source}" properties="intermine.properties" />
and the corresponding code in SlackNotify.java
public void setProperties(String propertiesFile) {
   projectProperties = new Properties();
   try {
     ClassLoader cl = SlackNotify.class.getClassLoader();
     InputStream is = cl.getResourceAsStream(propertiesFile);
      if (is == null) {
         throw new BuildException("Could not find file " + propertiesFile);
     }
     projectProperties.load(is);
     is.close();
 } catch (IOException e) {
     throw new BuildException("Failed to load :" + propertiesFile, e);
 }
}
This does not run since ./build/main is not in the classpath. If I
look at the variable cl in the debugger, I see that the only element
in the classpath is im-ant-tasks.jar. I can manually add the directory
to the classpath with
 AntClassLoader cl = (AntClassLoader) SlackNotify.class.getClassLoader();
cl.addPathElement(getProject().getBaseDir().toString()+"/build/main");
That's just wrong. It works, but I don't like this solution.
But, here's the thing, when I look at loadProperties in
org.intermine.metadata.Util, it has essentially the same code
  protected static Properties loadProperties(String filename) {
       Properties props = new NonOverrideableProperties();
       try {
           ClassLoader loader = Util.class.getClassLoader();
           InputStream is = loader.getResourceAsStream(filename);
           if (is == null) {
               LOG.error("Could not find file " + filename + " from " + loader);
               return null;
           }
           props.load(is);
           is.close();
       } catch (IOException e) {
           throw new RuntimeException("Failed to load :" + filename, e);
       }
       return props;
   }
This code is executed as part of the integration step as well. When I
look at the variable loader, it contains the element of the classpath
that I'm hoping to access. It certainly makes sense that
Integrate.java (or something it calls) has access to the properties
files since it uses that in establishing the db connections. As far as
I can tell, when the code for execute() in Integrate.java is called,
the classpath is still only im-ant-tasks.jar. But later the classpath
when Util.class is called includes the directories I'm looking for.
somehow, somewhere in task.xml or integrate.xml or something, there is
a mechanism for setting the classpath. And I assume some class is
doing the manipulation of the classpath. I had thought that it would
involve tossing in
  <path id="task.class.path">
     <pathelement location="${build.task.dir}"/>
  </path>
somewhere. But, where? That seems to be where I'm stuck. How do I set
the classpath in the xml files so that my code can find
intermine.properties?
Can someone give me a tip on setting up the classpath properly?
Thanks
Joe
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev



_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: silly ant question

justincc
On 04/05/16 21:21, Joe Carlson wrote:

>
>> On May 4, 2016, at 1:07 PM, [hidden email] <mailto:[hidden email]> wrote:
>>
>> Hi Joe,
>>
>> I don't think this is a silly question at all - I think InterMine's build system is convoluted and hard to pick through.
>>
>
> thanks. It’s not just me who thought the build system was tough.
>
>> In this case, I believe (though I could be wrong) that the task.class.path is only used in loading tasks to ant, not in their execution.  I believe that
>> setting up the classpath for execution involves invoking org.intermine.tasks.Dependencies, which is chiefly triggered via the internal -init-deps target in
>> library.xml.
>>
>> As you can see from integrate.xml, the "integrate" target depends on "-init-deps", which in integrate.xml then calls through to the imported library.init-deps
>>
>> The Dependencies class sets up a bunch of ant classpath properties via its execute() method, including project.execute.path, for instance, which is used as
>> the classpath in several places.  I suspect Util.loadProperties() by a task after this has been set up, though again I could be wrong.  Invoking your task may
>> require making sure that -init-deps (or maybe even just <dependencies>) has been triggered at some prior point.
>>
>> Exactly how are you defining the target that triggers <slack-notify>?
>
> this is probably where my problem is. As soon as I try to determine what is a target and what is a task and what is a typedef… my eyes glaze over. And it may
> not help that ‘integrate’ is both a target and a task.
>
> I thought I could get a new task into the system by adding to antlib.xml:
>         <typedef name="slack-notify" classname="org.intermine.task.SlackNotify”/>
>
> and add the task to the integrate target in integrate.xml:
>
>   <target name="integrate" depends="-init-properties, init, -init-deps, -init-integrate">
>      <integrate projectXml="../project.xml" source="${source}" action="${action}"/>
>      <!-- not standard. -->
>      <slack-notify source="${source}" properties="intermine.properties" />
>    </target>
>
> This passes intermine.properties into the setProperties routine. And it defines the source so that the notify messages captures that. This step does depend on
> -init-deps, so that part is good.
>
> It was mostly very simple to add a task. The one last bit about the class path has me stumped.

Having poked around, I believe now that you want to have your task code and definition in the intermine-integrate-main project, not in imbuild.

This is because the integrate task is actually performing a series of substeps (e.g. "retrieve" then "load") which invokes an entirely separate Ant instance
(you can see this in Integrate.performAction()).  At the Integrate level the classpath is not defined as you need it, whereas at the action (task) level it is
in imbuild/task.xml, which sets up task.class.path

I was wrong earlier about the task class path - this is used both to load the task and to invoke it.

To give my own test code, I have the TestTask class

package org.intermine.task;

import java.io.File;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;

public class TestTask extends Task {

     private File interminePropertiesFile;

     public void setIntermineProperties(File interminePropsFile) {
         this.interminePropertiesFile = interminePropsFile;
     }

     public void execute() throws BuildException {
         System.out.println("Hello World");
         ClassLoader cl = TestTask.class.getClassLoader();
         System.out.println("[" + cl.toString() + "]");
         System.out.println("interminePropertiesFile [" + interminePropertiesFile == null ? "null" : interminePropertiesFile.toString() + "]");
     }
}

in intermine-integrate-main/src/org/intermine/task/TestTask.java with the entry

<antlib>
     ...
     <typedef
         name="test-task"
         classname="org.intermine.task.TestTask"/>    
</antlib>

in the antlib.xml in that directory.  I then invoke this in the load target of source.xml with

   <!-- Load data from tgt items into production -->
   <target name="load" depends="init, -init-deps, -pre-load">
     <echo>
       Loading ${source.name} (${source.type}) tgt items into production DB
     </echo>
     <test-task/>
     <data-load integrationWriter="integration.production"
                source="os.${common.os.prefix}-translated"
                sourceName="${source.name}"
                sourceType="${source.type}"
                ignoreDuplicates="${ignore.duplicates}"
                allSources="${allsources.list}"/>
   </target>

(I'm only doing this before data-load for test purposes since I didn't want to keep purging the db).  You should be able to see that this gives TestTask the
full classpath from which intermine.properties should be avaiable.

There are other ways of going about this, such as passing in an intermineProperties XML attribute rather than relying on classpath, or doing some more work at
the parent integrate.xml level.  But this seems to fit in best with the current build structure.

-- Justin
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: silly ant question

joe carlson
Hi Justin,

thanks big! This is very useful. I had wondered about the task/subtask
structure of things. I had looked at this in the past but didn't realize
that it would have implications in the classpath settings. It certainly
makes sense - in retrospect.  And moving the task did the trick for me.

Joe

On 05/05/2016 08:58 AM, Justin Clark-Casey wrote:

> On 04/05/16 21:21, Joe Carlson wrote:
>>
>>> On May 4, 2016, at 1:07 PM, [hidden email]
>>> <mailto:[hidden email]> wrote:
>>>
>>> Hi Joe,
>>>
>>> I don't think this is a silly question at all - I think InterMine's
>>> build system is convoluted and hard to pick through.
>>>
>>
>> thanks. It’s not just me who thought the build system was tough.
>>
>>> In this case, I believe (though I could be wrong) that the
>>> task.class.path is only used in loading tasks to ant, not in their
>>> execution.  I believe that
>>> setting up the classpath for execution involves invoking
>>> org.intermine.tasks.Dependencies, which is chiefly triggered via the
>>> internal -init-deps target in
>>> library.xml.
>>>
>>> As you can see from integrate.xml, the "integrate" target depends on
>>> "-init-deps", which in integrate.xml then calls through to the
>>> imported library.init-deps
>>>
>>> The Dependencies class sets up a bunch of ant classpath properties
>>> via its execute() method, including project.execute.path, for
>>> instance, which is used as
>>> the classpath in several places.  I suspect Util.loadProperties() by
>>> a task after this has been set up, though again I could be wrong.  
>>> Invoking your task may
>>> require making sure that -init-deps (or maybe even just
>>> <dependencies>) has been triggered at some prior point.
>>>
>>> Exactly how are you defining the target that triggers <slack-notify>?
>>
>> this is probably where my problem is. As soon as I try to determine
>> what is a target and what is a task and what is a typedef… my eyes
>> glaze over. And it may
>> not help that ‘integrate’ is both a target and a task.
>>
>> I thought I could get a new task into the system by adding to
>> antlib.xml:
>>         <typedef name="slack-notify"
>> classname="org.intermine.task.SlackNotify”/>
>>
>> and add the task to the integrate target in integrate.xml:
>>
>>   <target name="integrate" depends="-init-properties, init,
>> -init-deps, -init-integrate">
>>      <integrate projectXml="../project.xml" source="${source}"
>> action="${action}"/>
>>      <!-- not standard. -->
>>      <slack-notify source="${source}"
>> properties="intermine.properties" />
>>    </target>
>>
>> This passes intermine.properties into the setProperties routine. And
>> it defines the source so that the notify messages captures that. This
>> step does depend on
>> -init-deps, so that part is good.
>>
>> It was mostly very simple to add a task. The one last bit about the
>> class path has me stumped.
>
> Having poked around, I believe now that you want to have your task
> code and definition in the intermine-integrate-main project, not in
> imbuild.
>
> This is because the integrate task is actually performing a series of
> substeps (e.g. "retrieve" then "load") which invokes an entirely
> separate Ant instance (you can see this in
> Integrate.performAction()).  At the Integrate level the classpath is
> not defined as you need it, whereas at the action (task) level it is
> in imbuild/task.xml, which sets up task.class.path
>
> I was wrong earlier about the task class path - this is used both to
> load the task and to invoke it.
>
> To give my own test code, I have the TestTask class
>
> package org.intermine.task;
>
> import java.io.File;
>
> import org.apache.tools.ant.BuildException;
> import org.apache.tools.ant.Task;
>
> public class TestTask extends Task {
>
>     private File interminePropertiesFile;
>
>     public void setIntermineProperties(File interminePropsFile) {
>         this.interminePropertiesFile = interminePropsFile;
>     }
>
>     public void execute() throws BuildException {
>         System.out.println("Hello World");
>         ClassLoader cl = TestTask.class.getClassLoader();
>         System.out.println("[" + cl.toString() + "]");
>         System.out.println("interminePropertiesFile [" +
> interminePropertiesFile == null ? "null" :
> interminePropertiesFile.toString() + "]");
>     }
> }
>
> in intermine-integrate-main/src/org/intermine/task/TestTask.java with
> the entry
>
> <antlib>
>     ...
>     <typedef
>         name="test-task"
>         classname="org.intermine.task.TestTask"/>
> </antlib>
>
> in the antlib.xml in that directory.  I then invoke this in the load
> target of source.xml with
>
>   <!-- Load data from tgt items into production -->
>   <target name="load" depends="init, -init-deps, -pre-load">
>     <echo>
>       Loading ${source.name} (${source.type}) tgt items into
> production DB
>     </echo>
>     <test-task/>
>     <data-load integrationWriter="integration.production"
>                source="os.${common.os.prefix}-translated"
>                sourceName="${source.name}"
>                sourceType="${source.type}"
>                ignoreDuplicates="${ignore.duplicates}"
>                allSources="${allsources.list}"/>
>   </target>
>
> (I'm only doing this before data-load for test purposes since I didn't
> want to keep purging the db).  You should be able to see that this
> gives TestTask the full classpath from which intermine.properties
> should be avaiable.
>
> There are other ways of going about this, such as passing in an
> intermineProperties XML attribute rather than relying on classpath, or
> doing some more work at the parent integrate.xml level.  But this
> seems to fit in best with the current build structure.
>
> -- Justin

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Loading...