Again on display on external UCSC

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Again on display on external UCSC

Davide Cittaro
Hi again... I still have problems in visualizing on external UCSC sites using the display_application... I've implemented bigWig data format, and it works (not really the sniffer but who cares at the moment?).... What is important is that the display_applications gives me a link to the genome browser that is correct:


which points to


which has this link in hgt.customText


which says

track type=bigWig name="wgEncodeYaleChIPseqRawDataRep2Helas3Pol2_fs.bw" bigDataUrl=http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b9b902f807f2/ucsc_bigwig/campus/090e9746a4403c88/data/galaxy_cac7b9b902f807f2.bw db=hg18

Everything is correct, apparently...
Needles to say, the bigWig file is fully operational meaning that:
1- if I download the bigDataUrl to another machine and show that file through another apache, the genome browser can read it
2- the original file can be seen and loaded.

BTW, the genome browser complains:


That probably means that there's something between the genome browser and the bigWig file (galaxy?) which passes wrong data/headers or possibly is passing something *before* the actual bigWig data stream...
As I have issues with BAM files too, I believe the cause may be the same... 

Any help/hint/debug strategy is really appreciated!

d
/*
Davide Cittaro

Cogentech - Consortium for Genomic Technologies
via adamello, 16
20139 Milano
Italy

tel.: +39(02)574303007
*/




_______________________________________________
galaxy-dev mailing list
[hidden email]
http://lists.bx.psu.edu/listinfo/galaxy-dev
Reply | Threaded
Open this post in threaded view
|

Help with tcp traffic (was Again on display on external UCSC)

Davide Cittaro
My attempts to understand why a bigwig file cannot be loaded by UCSC through galaxy's display_application ended up with tcpdump... If anybody would like to help me understanding what's wrong (and different) here I would really appreciate it.
If needed I can make the tcpdupms available for analysis with wireshark...

[ TCP Stream for a bigWig served outside galaxy]

[stream 1: after setting hgt.customText in hgTracks equal to a path with bigWig track definition]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:50:29 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Fri, 11 Jun 2010 13:37:28 GMT
ETag: "7140-9f-488c13d753600"
Accept-Ranges: bytes
Content-Length: 159
Connection: close
Content-Type: text/plain

track type=bigWig name="therightone.bw" bigDataUrl=http://host001.instruments.ifom-ieo-campus.it/ga-data/galaxy_dist/database/files/000/dataset_46.dat db=hg18

[stream 2 ]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:50:29 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT
ETag: "9f-42e5586-488ae729691c0"
Accept-Ranges: bytes
Content-Length: 70145414
Connection: close
Content-Type: chemical/x-mopac-input

[stream 3]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:50:36 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT
ETag: "9f-42e5586-488ae729691c0"
Accept-Ranges: bytes
Content-Length: 70145414
Connection: close
Content-Type: chemical/x-mopac-input

[stream 4: I believe this is after I ask for a different region on chromosome]

HTTP/1.1 206 Partial Content
Date: Fri, 11 Jun 2010 13:50:37 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT
ETag: "9f-42e5586-488ae729691c0"
Accept-Ranges: bytes
Content-Length: 9819526
Content-Range: bytes 60325888-70145413/70145414
Connection: close
Content-Type: chemical/x-mopac-input

%.:....3...@...=>.} [cut, note that the bigWig data are not from the beginning of file]

[stream 5]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:50:40 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT
ETag: "9f-42e5586-488ae729691c0"
Accept-Ranges: bytes
Content-Length: 70145414
Connection: close
Content-Type: chemical/x-mopac-input


[ TCP Stream for the same  bigWig served within galaxy]

[stream 1: after hgt.customText is set to the path served by galaxy]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:46:29 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Fri, 11 Jun 2010 13:36:56 GMT
ETag: "713f-f7-488c13b8cee00"
Accept-Ranges: bytes
Content-Length: 247
Connection: close
Content-Type: text/plain

track type=bigWig name="wgEncodeYaleChIPseqRawDataRep2Helas3Pol2_fs.bw" bigDataUrl=http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b9b902f807f2/ucsc_bigwig/campus/090e9746a4403c88/data/galaxy_cac7b9b902f807f2.bw db=hg18

[stream 2]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:46:29 GMT
Server: PasteWSGIServer/0.5 Python/2.6.5
content-length: 70145414
content-type: chemical/x-mopac-input
Set-Cookie: galaxysession=eb142648ac45b770ddabc752e2cd511e21664b5a654cabfbe79569d4ede184379f55c5dbc9000304; expires=Thu, 09-Sep-2010 15:46:29 GMT; Max-Age=7776000; Path=/; Version=1
Connection: close


HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:46:29 GMT
Server: PasteWSGIServer/0.5 Python/2.6.5
content-length: 70145414
content-type: chemical/x-mopac-input
Set-Cookie: galaxysession=eb142648ac45b770233699f0ab65857e995c6e1b17d8e57d15b91a28b7b680af4cf210ffa8e6cc2a; expires=Thu, 09-Sep-2010 15:46:29 GMT; Max-Age=7776000; Path=/; Version=1
Connection: close

&.......(......... [cut: note that bigWig data are served from the beginning, note that http headers are different]

[stream 4: another attept to navigate on a different region of the chromosome]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:46:30 GMT
Server: PasteWSGIServer/0.5 Python/2.6.5
content-length: 70145414
content-type: chemical/x-mopac-input
Set-Cookie: galaxysession=eb142648ac45b7709206aaeef08ccecdd9d6070c5f8aed59d9ddd2079322251934ece94b298127df; expires=Thu, 09-Sep-2010 15:46:30 GMT; Max-Age=7776000; Path=/; Version=1
Connection: close

&.......(...............$. [cut: again, the bigWig data are sent entirely from the beginning]




/*
Davide Cittaro

Cogentech - Consortium for Genomic Technologies
via adamello, 16
20139 Milano
Italy

tel.: +39(02)574303007
*/




_______________________________________________
galaxy-dev mailing list
[hidden email]
http://lists.bx.psu.edu/listinfo/galaxy-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with tcp traffic (was Again on display on external UCSC)

Davide Cittaro
Looking at the headers, the big difference seems that galaxy webserver doesn't allow for "partial content"...

On Jun 14, 2010, at 12:04 PM, Davide Cittaro wrote:
[stream 2 (working)]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:50:29 GMT
Server: Apache/2.2.8 (Ubuntu)
Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT
ETag: "9f-42e5586-488ae729691c0"
Accept-Ranges: bytes
Content-Length: 70145414
Connection: close
Content-Type: chemical/x-mopac-input

[stream 2 (not working)]

HTTP/1.1 200 OK
Date: Fri, 11 Jun 2010 13:46:29 GMT
Server: PasteWSGIServer/0.5 Python/2.6.5
content-length: 70145414
content-type: chemical/x-mopac-input
Set-Cookie: galaxysession=eb142648ac45b770ddabc752e2cd511e21664b5a654cabfbe79569d4ede184379f55c5dbc9000304; expires=Thu, 09-Sep-2010 15:46:29 GMT; Max-Age=7776000; Path=/; Version=1
Connection: close

Actually no "Accept-Ranges: bytes" nor the entity tag are present, I guess the hgTracks binary can't send a request for a specific section of the file and receives the whole stream in a single request. Looking at the w3c pages:

10.2.7 206 Partial Content

The server has fulfilled the partial GET request for the resource. The request MUST have included a Range header field (section 14.35) indicating the desired range, and MAY have included an If-Range header field (section 14.27) to make the request conditional.

The response MUST include the following header fields:

      - Either a Content-Range header field (section 14.16) indicating
        the range included with this response, or a multipart/byteranges
        Content-Type including Content-Range fields for each part. If a
        Content-Length header field is present in the response, its
        value MUST match the actual number of OCTETs transmitted in the
        message-body.
      - Date
      - ETag and/or Content-Location, if the header would have been sent
        in a 200 response to the same request
      - Expires, Cache-Control, and/or Vary, if the field-value might
        differ from that sent in any previous response for the same
        variant

If the 206 response is the result of an If-Range request that used a strong cache validator (see section 13.3.3), the response SHOULD NOT include other entity-headers. If the response is the result of an If-Range request that used a weak validator, the response MUST NOT include other entity-headers; this prevents inconsistencies between cached entity-bodies and updated headers. Otherwise, the response MUST include all of the entity-headers that would have been returned with a 200 (OK) response to the same request.

A cache MUST NOT combine a 206 response with other previously cached content if the ETag or Last-Modified headers do not match exactly, see 13.5.4.

A cache that does not support the Range and Content-Range headers MUST NOT cache 206 (Partial) responses.


The header fields listed above are not included when galaxy tries to serve the bigWig file. Is there a way to modify this behavior (possibly with the new datatype definition...).

Thanks

d

/*
Davide Cittaro

Cogentech - Consortium for Genomic Technologies
via adamello, 16
20139 Milano
Italy

tel.: +39(02)574303007
*/




_______________________________________________
galaxy-dev mailing list
[hidden email]
http://lists.bx.psu.edu/listinfo/galaxy-dev