upload with python client not working

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

upload with python client not working

Chris
Hi all,

I'm in the process of buliding some uploading functionality using the python-apollo client. I have the following zip file that I am attempting to upload.

Archive:  data/data.zip
 
Length      Date    Time    Name
---------  ---------- -----   ----
       
0  03-02-2020 14:34   seq/
     
993  03-02-2020 14:34   seq/refSeqs.json
 
29752363  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna
     
256  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna.fai
     
533  03-02-2020 14:34   trackList.json
       
0  03-02-2020 14:34   tracks.conf
---------                     -------
 
29754145                     6 files

Using this code:

wa = ApolloInstance('http://localhost:11000', credentials["webapollo"]["username"], credentials["webapollo"]["password"])
...
wa
.remote.add_organism("Fumigatus",
                open
("data/data.zip", "rb"),
                blatdb
="Aspergillus_fumigatus_genomic.fna.2bit",
                genus
="Aspergillus",
                species
="fumigatus")

I get the following json response from apollo:

{'error': '/data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)'}

If I look on the server, this file is indeed missing, but I have no idea why. I am able to create the organism manually from the apollo interface using the exploded zip data. Is this a known issue? The server is remote to the code I'm executing, running in docker (quay.io/gmod/apollo:2.5.0).

Here are the server logs:

2020-03-02 20:08:49,543 [http-nio-8080-exec-8] INFO  apollo.OrganismController  - Adding organism with SEQUENCE [commonName:Fumigatus, password:password, controller:organism, genus:Aspergillus, species:fumigatus, clientToken:ignore, format:null, action:addOrganismWithSequence, organismData:org.springframework.web.multipart.commons.CommonsMultipartFile@10839313, searchDatabaseData:org.springframework.web.multipart.commons.CommonsMultipartFile@108b80b2, username:admin@local.host]
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus does not exist
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be read
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be written to
2020-03-02 20:08:49,555 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Successfully created directory /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,557 [http-nio-8080-exec-8] DEBUG apollo.FileService  - initial location: /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,803 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file trackList.json vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,806 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file tracks.conf vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,808 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Adding Fumigatus with directory: /data/temporary/apollo_data/110-Fumigatus
java
.io.FileNotFoundException: /data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)

Thanks,
Chris

--
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: upload with python client not working

nathandunn
Chris,

There was a bug we fixed a few weeks ago where symlinks were not being handled properly:


Could you try this with gmod/apollo:latest ? 

Thanks,

Nathan

On Mar 2, 2020, at 12:22 PM, Chris <[hidden email]> wrote:

Hi all,

I'm in the process of buliding some uploading functionality using the python-apollo client. I have the following zip file that I am attempting to upload.

Archive:  data/data.zip
 
Length      Date    Time    Name
---------  ---------- -----   ----
       
0  03-02-2020 14:34   seq/
     
993  03-02-2020 14:34   seq/refSeqs.json
 
29752363  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna
     
256  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna.fai
     
533  03-02-2020 14:34   trackList.json
       
0  03-02-2020 14:34   tracks.conf
---------                     -------
 
29754145                     6 files

Using this code:

wa = ApolloInstance('http://localhost:11000', credentials["webapollo"]["username"], credentials["webapollo"]["password"])
...
wa
.remote.add_organism("Fumigatus",
                open
("data/data.zip", "rb"),
                blatdb
="Aspergillus_fumigatus_genomic.fna.2bit",
                genus
="Aspergillus",
                species
="fumigatus")

I get the following json response from apollo:

{'error': '/data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)'}

If I look on the server, this file is indeed missing, but I have no idea why. I am able to create the organism manually from the apollo interface using the exploded zip data. Is this a known issue? The server is remote to the code I'm executing, running in docker (quay.io/gmod/apollo:2.5.0).

Here are the server logs:

2020-03-02 20:08:49,543 [http-nio-8080-exec-8] INFO  apollo.OrganismController  - Adding organism with SEQUENCE [commonName:Fumigatus, password:password, controller:organism, genus:Aspergillus, species:fumigatus, clientToken:ignore, format:null, action:addOrganismWithSequence, organismData:org.springframework.web.multipart.commons.CommonsMultipartFile@10839313, searchDatabaseData:org.springframework.web.multipart.commons.CommonsMultipartFile@108b80b2, username:admin@local.host]
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus does not exist
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be read
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be written to
2020-03-02 20:08:49,555 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Successfully created directory /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,557 [http-nio-8080-exec-8] DEBUG apollo.FileService  - initial location: /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,803 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file trackList.json vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,806 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file tracks.conf vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,808 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Adding Fumigatus with directory: /data/temporary/apollo_data/110-Fumigatus
java
.io.FileNotFoundException: /data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)

Thanks,
Chris

--
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: upload with python client not working

Chris
Hi Nathan,

Thanks. I updated to the new docker image, but the issue remains.

Chris

— 
Christopher J. Friedline, PhD
Scientific Consultant
Diamond Age Data Science


On Mon, Mar 2, 2020 at 4:43 PM Nathan Dunn <[hidden email]> wrote:
Chris,

There was a bug we fixed a few weeks ago where symlinks were not being handled properly:


Could you try this with gmod/apollo:latest ? 

Thanks,

Nathan

On Mar 2, 2020, at 12:22 PM, Chris <[hidden email]> wrote:

Hi all,

I'm in the process of buliding some uploading functionality using the python-apollo client. I have the following zip file that I am attempting to upload.

Archive:  data/data.zip
 
Length      Date    Time    Name
---------  ---------- -----   ----
       
0  03-02-2020 14:34   seq/
     
993  03-02-2020 14:34   seq/refSeqs.json
 
29752363  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna
     
256  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna.fai
     
533  03-02-2020 14:34   trackList.json
       
0  03-02-2020 14:34   tracks.conf
---------                     -------
 
29754145                     6 files

Using this code:

wa = ApolloInstance('http://localhost:11000', credentials["webapollo"]["username"], credentials["webapollo"]["password"])
...
wa
.remote.add_organism("Fumigatus",
                open
("data/data.zip", "rb"),
                blatdb
="Aspergillus_fumigatus_genomic.fna.2bit",
                genus
="Aspergillus",
                species
="fumigatus")

I get the following json response from apollo:

{'error': '/data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)'}

If I look on the server, this file is indeed missing, but I have no idea why. I am able to create the organism manually from the apollo interface using the exploded zip data. Is this a known issue? The server is remote to the code I'm executing, running in docker (quay.io/gmod/apollo:2.5.0).

Here are the server logs:

2020-03-02 20:08:49,543 [http-nio-8080-exec-8] INFO  apollo.OrganismController  - Adding organism with SEQUENCE [commonName:Fumigatus, password:password, controller:organism, genus:Aspergillus, species:fumigatus, clientToken:ignore, format:null, action:addOrganismWithSequence, organismData:org.springframework.web.multipart.commons.CommonsMultipartFile@10839313, searchDatabaseData:org.springframework.web.multipart.commons.CommonsMultipartFile@108b80b2, username:admin@local.host]
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus does not exist
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be read
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be written to
2020-03-02 20:08:49,555 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Successfully created directory /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,557 [http-nio-8080-exec-8] DEBUG apollo.FileService  - initial location: /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,803 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file trackList.json vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,806 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file tracks.conf vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,808 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Adding Fumigatus with directory: /data/temporary/apollo_data/110-Fumigatus
java
.io.FileNotFoundException: /data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)

Thanks,
Chris

--
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: upload with python client not working

nathandunn

Okay.  I’ll take another look. 

Nathan


On Mar 2, 2020, at 1:54 PM, Chris Friedline <[hidden email]> wrote:

Hi Nathan,

Thanks. I updated to the new docker image, but the issue remains.

Chris

— 
Christopher J. Friedline, PhD
Scientific Consultant
Diamond Age Data Science


On Mon, Mar 2, 2020 at 4:43 PM Nathan Dunn <[hidden email]> wrote:
Chris,

There was a bug we fixed a few weeks ago where symlinks were not being handled properly:


Could you try this with gmod/apollo:latest ? 

Thanks,

Nathan

On Mar 2, 2020, at 12:22 PM, Chris <[hidden email]> wrote:

Hi all,

I'm in the process of buliding some uploading functionality using the python-apollo client. I have the following zip file that I am attempting to upload.

Archive:  data/data.zip
 
Length      Date    Time    Name
---------  ---------- -----   ----
       
0  03-02-2020 14:34   seq/
     
993  03-02-2020 14:34   seq/refSeqs.json
 
29752363  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna
     
256  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna.fai
     
533  03-02-2020 14:34   trackList.json
       
0  03-02-2020 14:34   tracks.conf
---------                     -------
 
29754145                     6 files

Using this code:

wa = ApolloInstance('http://localhost:11000', credentials["webapollo"]["username"], credentials["webapollo"]["password"])
...
wa
.remote.add_organism("Fumigatus",
                open
("data/data.zip", "rb"),
                blatdb
="Aspergillus_fumigatus_genomic.fna.2bit",
                genus
="Aspergillus",
                species
="fumigatus")

I get the following json response from apollo:

{'error': '/data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)'}

If I look on the server, this file is indeed missing, but I have no idea why. I am able to create the organism manually from the apollo interface using the exploded zip data. Is this a known issue? The server is remote to the code I'm executing, running in docker (quay.io/gmod/apollo:2.5.0).

Here are the server logs:

2020-03-02 20:08:49,543 [http-nio-8080-exec-8] INFO  apollo.OrganismController  - Adding organism with SEQUENCE [commonName:Fumigatus, password:password, controller:organism, genus:Aspergillus, species:fumigatus, clientToken:ignore, format:null, action:addOrganismWithSequence, organismData:org.springframework.web.multipart.commons.CommonsMultipartFile@10839313, searchDatabaseData:org.springframework.web.multipart.commons.CommonsMultipartFile@108b80b2, username:admin@local.host]
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus does not exist
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be read
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be written to
2020-03-02 20:08:49,555 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Successfully created directory /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,557 [http-nio-8080-exec-8] DEBUG apollo.FileService  - initial location: /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,803 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file trackList.json vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,806 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file tracks.conf vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,808 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Adding Fumigatus with directory: /data/temporary/apollo_data/110-Fumigatus
java
.io.FileNotFoundException: /data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)

Thanks,
Chris


--
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: upload with python client not working

nathandunn
In reply to this post by Chris

I created a PR to hopefully fix this issue:


Will let you know when I get through this.  Hopefully by this week or the end of next week.

Cheers,

Nathan


On Mar 2, 2020, at 1:54 PM, Chris Friedline <[hidden email]> wrote:

Hi Nathan,

Thanks. I updated to the new docker image, but the issue remains.

Chris

— 
Christopher J. Friedline, PhD
Scientific Consultant
Diamond Age Data Science


On Mon, Mar 2, 2020 at 4:43 PM Nathan Dunn <[hidden email]> wrote:
Chris,

There was a bug we fixed a few weeks ago where symlinks were not being handled properly:


Could you try this with gmod/apollo:latest ? 

Thanks,

Nathan

On Mar 2, 2020, at 12:22 PM, Chris <[hidden email]> wrote:

Hi all,

I'm in the process of buliding some uploading functionality using the python-apollo client. I have the following zip file that I am attempting to upload.

Archive:  data/data.zip
 
Length      Date    Time    Name
---------  ---------- -----   ----
       
0  03-02-2020 14:34   seq/
     
993  03-02-2020 14:34   seq/refSeqs.json
 
29752363  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna
     
256  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna.fai
     
533  03-02-2020 14:34   trackList.json
       
0  03-02-2020 14:34   tracks.conf
---------                     -------
 
29754145                     6 files

Using this code:

wa = ApolloInstance('http://localhost:11000', credentials["webapollo"]["username"], credentials["webapollo"]["password"])
...
wa
.remote.add_organism("Fumigatus",
                open
("data/data.zip", "rb"),
                blatdb
="Aspergillus_fumigatus_genomic.fna.2bit",
                genus
="Aspergillus",
                species
="fumigatus")

I get the following json response from apollo:

{'error': '/data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)'}

If I look on the server, this file is indeed missing, but I have no idea why. I am able to create the organism manually from the apollo interface using the exploded zip data. Is this a known issue? The server is remote to the code I'm executing, running in docker (quay.io/gmod/apollo:2.5.0).

Here are the server logs:

2020-03-02 20:08:49,543 [http-nio-8080-exec-8] INFO  apollo.OrganismController  - Adding organism with SEQUENCE [commonName:Fumigatus, password:password, controller:organism, genus:Aspergillus, species:fumigatus, clientToken:ignore, format:null, action:addOrganismWithSequence, organismData:org.springframework.web.multipart.commons.CommonsMultipartFile@10839313, searchDatabaseData:org.springframework.web.multipart.commons.CommonsMultipartFile@108b80b2, username:admin@local.host]
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus does not exist
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be read
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be written to
2020-03-02 20:08:49,555 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Successfully created directory /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,557 [http-nio-8080-exec-8] DEBUG apollo.FileService  - initial location: /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,803 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file trackList.json vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,806 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file tracks.conf vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,808 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Adding Fumigatus with directory: /data/temporary/apollo_data/110-Fumigatus
java
.io.FileNotFoundException: /data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)

Thanks,
Chris


--
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: upload with python client not working

nathandunn
In reply to this post by Chris

This should be fixed in the PR. 

The docker is currently rebuilding on quay (takes about 30 minutes): https://quay.io/repository/gmod/apollo?tab=builds

I’m sure it will be up on docker hub in the next few hours. 

Let me know if you encounter any further problems. 

Nathan



On Mar 2, 2020, at 1:54 PM, Chris Friedline <[hidden email]> wrote:

Hi Nathan,

Thanks. I updated to the new docker image, but the issue remains.

Chris

— 
Christopher J. Friedline, PhD
Scientific Consultant
Diamond Age Data Science


On Mon, Mar 2, 2020 at 4:43 PM Nathan Dunn <[hidden email]> wrote:
Chris,

There was a bug we fixed a few weeks ago where symlinks were not being handled properly:


Could you try this with gmod/apollo:latest ? 

Thanks,

Nathan

On Mar 2, 2020, at 12:22 PM, Chris <[hidden email]> wrote:

Hi all,

I'm in the process of buliding some uploading functionality using the python-apollo client. I have the following zip file that I am attempting to upload.

Archive:  data/data.zip
 
Length      Date    Time    Name
---------  ---------- -----   ----
       
0  03-02-2020 14:34   seq/
     
993  03-02-2020 14:34   seq/refSeqs.json
 
29752363  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna
     
256  03-02-2020 14:34   seq/Aspergillus_fumigatus_genomic.fna.fai
     
533  03-02-2020 14:34   trackList.json
       
0  03-02-2020 14:34   tracks.conf
---------                     -------
 
29754145                     6 files

Using this code:

wa = ApolloInstance('http://localhost:11000', credentials["webapollo"]["username"], credentials["webapollo"]["password"])
...
wa
.remote.add_organism("Fumigatus",
                open
("data/data.zip", "rb"),
                blatdb
="Aspergillus_fumigatus_genomic.fna.2bit",
                genus
="Aspergillus",
                species
="fumigatus")

I get the following json response from apollo:

{'error': '/data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)'}

If I look on the server, this file is indeed missing, but I have no idea why. I am able to create the organism manually from the apollo interface using the exploded zip data. Is this a known issue? The server is remote to the code I'm executing, running in docker (quay.io/gmod/apollo:2.5.0).

Here are the server logs:

2020-03-02 20:08:49,543 [http-nio-8080-exec-8] INFO  apollo.OrganismController  - Adding organism with SEQUENCE [commonName:Fumigatus, password:password, controller:organism, genus:Aspergillus, species:fumigatus, clientToken:ignore, format:null, action:addOrganismWithSequence, organismData:org.springframework.web.multipart.commons.CommonsMultipartFile@10839313, searchDatabaseData:org.springframework.web.multipart.commons.CommonsMultipartFile@108b80b2, username:admin@local.host]
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus does not exist
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be read
2020-03-02 20:08:49,554 [http-nio-8080-exec-8] WARN  apollo.TrackService  - File /data/temporary/apollo_data/110-Fumigatus can not be written to
2020-03-02 20:08:49,555 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Successfully created directory /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,557 [http-nio-8080-exec-8] DEBUG apollo.FileService  - initial location: /data/temporary/apollo_data/110-Fumigatus
2020-03-02 20:08:49,803 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file trackList.json vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,806 [http-nio-8080-exec-8] ERROR apollo.FileService  - Problem decrompression file tracks.conf vs seq/
java
.io.IOException: File is outside extraction target directory.
    at org
.bbop.apollo.FileService.$tt__validateFileName(FileService.groovy:338)
    at org
.bbop.apollo.FileService.$tt__decompressZipArchive(FileService.groovy:69)
    at org
.bbop.apollo.FileService.$tt__decompress(FileService.groovy:33)
    at org
.bbop.apollo.OrganismController.$tt__addOrganismWithSequence(OrganismController.groovy:323)
    at grails
.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails
.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org
.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org
.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org
.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org
.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org
.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java
.lang.Thread.run(Thread.java:748)
2020-03-02 20:08:49,808 [http-nio-8080-exec-8] DEBUG apollo.OrganismController  - Adding Fumigatus with directory: /data/temporary/apollo_data/110-Fumigatus
java
.io.FileNotFoundException: /data/temporary/apollo_data/110-Fumigatus/trackList.json (No such file or directory)

Thanks,
Chris


--
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].