Upload is successful, user sees: "Uploaded file The server returned an error during upload"

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Upload is successful, user sees: "Uploaded file The server returned an error during upload"

Kevin Schaper-3
Hi,

While doing pre-release testing our new GBrowse2 instance, we found that .wig file uploads claim to fail...but then the track appears a short time later.  

We're currently just using regular cgi, I have the Timeout set at 1200.  (before I had set it, I would see timeout messages in the apache error log, though, it still showed the same error message while eventually succeeding)

What I found was that this message seems to appear at 90 seconds when uploading from a file, and 60 seconds when uploading from a URL.  From the lines being printed on the screen, it doesn't seem to do it after the same amount of data has been transferred.  (~1M lines for url, ~1.3M lines for file) - so it seems more like a time thing than a size thing.

Watching the post & response traffic with firebug, it doesn't look like anything comes from the server side to say that it's failing.  Does the front-end code have automatic timeouts to assume failure after a set amount of time?

thanks,
Kevin
  



 

------------------------------------------------------------------------------
Android apps run on BlackBerry 10
Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
Now with support for Jelly Bean, Bluetooth, Mapview and more.
Get your Android app in front of a whole new audience.  Start now.
http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: Upload is successful, user sees: "Uploaded file The server returned an error during upload"

Timothy Parnell
Hi Kevin,

I’m certainly not an expert on CGI, but I do know that gbrowse itself has its own timeout settings, which may be what you are running into.

In your main GBrowse.conf (usually in /etc/gbrowse2), there are these settings
slave_timeout          = 45
global_timeout         = 60
search_timeout         = 15

which I assume are seconds for the relative timeout. Unfortunately, no specific timeout for file uploads.

If you use fastcgi, there is another location in the gbrowse specific apache conf that controls various timeouts as well, including busy/idle or io.

So, yeah, I guess it may not be too surprising that the main process times out for the main display, but that the spawned process dealing with the file upload will continue in the background until finished, resulting in your wig file eventually showing up. I typically use fastcgi, with fastcgi timeouts of 3600 (the default?), and I do not see the timeouts when I upload large files.

I hope that helps,
Tim

On Feb 12, 2014, at 11:56 AM, Kevin Schaper <[hidden email]> wrote:

> Hi,
>
> While doing pre-release testing our new GBrowse2 instance, we found that .wig file uploads claim to fail...but then the track appears a short time later.  
>
> We're currently just using regular cgi, I have the Timeout set at 1200.  (before I had set it, I would see timeout messages in the apache error log, though, it still showed the same error message while eventually succeeding)
>
> What I found was that this message seems to appear at 90 seconds when uploading from a file, and 60 seconds when uploading from a URL.  From the lines being printed on the screen, it doesn't seem to do it after the same amount of data has been transferred.  (~1M lines for url, ~1.3M lines for file) - so it seems more like a time thing than a size thing.
>
> Watching the post & response traffic with firebug, it doesn't look like anything comes from the server side to say that it's failing.  Does the front-end code have automatic timeouts to assume failure after a set amount of time?
>
> thanks,
> Kevin
>  
>
>
>
>  
> ------------------------------------------------------------------------------
> Android apps run on BlackBerry 10
> Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
> Now with support for Jelly Bean, Bluetooth, Mapview and more.
> Get your Android app in front of a whole new audience.  Start now.
> http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk_______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse


------------------------------------------------------------------------------
Android apps run on BlackBerry 10
Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
Now with support for Jelly Bean, Bluetooth, Mapview and more.
Get your Android app in front of a whole new audience.  Start now.
http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: Upload is successful, user sees: "Uploaded file The server returned an error during upload"

Kevin Schaper-3
Hi Tim, 

thanks for helping me out again!

Once we switched to fcgid, and I set the timeouts to 1200 seconds, I was able to upload .wig files successfully and didn't see timeouts.

We have a 322M bed file though, and it still times out even with a 20 minute timeout, I'm testing it with a 1HR timeout, but I don't think that's something that will actually fly with our sysadmin. 

Do other public GBrowse sites allow uploads?  does everyone need to set ridiculous apache timeouts?  

My best guess is that the bed file may be slow because it's using SQLite as the upload database, but I'm also a little leery of taking on the extra MySQL management we would have if uploaded tracks went into the database.  

Thanks,
Kevin







On Wed, Feb 12, 2014 at 1:14 PM, Timothy Parnell <[hidden email]> wrote:
Hi Kevin,

I’m certainly not an expert on CGI, but I do know that gbrowse itself has its own timeout settings, which may be what you are running into.

In your main GBrowse.conf (usually in /etc/gbrowse2), there are these settings
slave_timeout          = 45
global_timeout         = 60
search_timeout         = 15

which I assume are seconds for the relative timeout. Unfortunately, no specific timeout for file uploads.

If you use fastcgi, there is another location in the gbrowse specific apache conf that controls various timeouts as well, including busy/idle or io.

So, yeah, I guess it may not be too surprising that the main process times out for the main display, but that the spawned process dealing with the file upload will continue in the background until finished, resulting in your wig file eventually showing up. I typically use fastcgi, with fastcgi timeouts of 3600 (the default?), and I do not see the timeouts when I upload large files.

I hope that helps,
Tim

On Feb 12, 2014, at 11:56 AM, Kevin Schaper <[hidden email]> wrote:

> Hi,
>
> While doing pre-release testing our new GBrowse2 instance, we found that .wig file uploads claim to fail...but then the track appears a short time later.
>
> We're currently just using regular cgi, I have the Timeout set at 1200.  (before I had set it, I would see timeout messages in the apache error log, though, it still showed the same error message while eventually succeeding)
>
> What I found was that this message seems to appear at 90 seconds when uploading from a file, and 60 seconds when uploading from a URL.  From the lines being printed on the screen, it doesn't seem to do it after the same amount of data has been transferred.  (~1M lines for url, ~1.3M lines for file) - so it seems more like a time thing than a size thing.
>
> Watching the post & response traffic with firebug, it doesn't look like anything comes from the server side to say that it's failing.  Does the front-end code have automatic timeouts to assume failure after a set amount of time?
>
> thanks,
> Kevin
>
>
>
>
>
> ------------------------------------------------------------------------------
> Android apps run on BlackBerry 10
> Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
> Now with support for Jelly Bean, Bluetooth, Mapview and more.
> Get your Android app in front of a whole new audience.  Start now.
> http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk_______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse



------------------------------------------------------------------------------
Android apps run on BlackBerry 10
Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
Now with support for Jelly Bean, Bluetooth, Mapview and more.
Get your Android app in front of a whole new audience.  Start now.
http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: Upload is successful, user sees: "Uploaded file The server returned an error during upload"

Timothy Parnell
Hi Kevin,

I’m not surprised that a bed file of that enormity would time out. You’re right, it’s probably the SQLite choking, considering it does not use any sort of fast loading (I can’t tell from a brief look at the code) and SQLite really slows down a lot when you get into very large imports.

This is really a case where the bed file should be converted into a bigBed file prior to loading, as that would be nearly instant to process after uploading. I had committed a bigBed uploader to the latest GBrowse version, so if you can convince users to change formats, that would be superior.

I guess user education would be the best route. Encourage uploads only of binary formats (bigWig, bigBed, bam) which don’t suck up server time converting text to other formats. Small text gff and bed files would be fine - I wish we could set a limit there for size and throw an error for text files that exceed a threshold.

Tim


On Feb 14, 2014, at 12:24 PM, Kevin Schaper <[hidden email]<mailto:[hidden email]>> wrote:

Hi Tim,

thanks for helping me out again!

Once we switched to fcgid, and I set the timeouts to 1200 seconds, I was able to upload .wig files successfully and didn't see timeouts.

We have a 322M bed file though, and it still times out even with a 20 minute timeout, I'm testing it with a 1HR timeout, but I don't think that's something that will actually fly with our sysadmin.

Do other public GBrowse sites allow uploads?  does everyone need to set ridiculous apache timeouts?

My best guess is that the bed file may be slow because it's using SQLite as the upload database, but I'm also a little leery of taking on the extra MySQL management we would have if uploaded tracks went into the database.

Thanks,
Kevin







On Wed, Feb 12, 2014 at 1:14 PM, Timothy Parnell <[hidden email]<mailto:[hidden email]>> wrote:
Hi Kevin,

I’m certainly not an expert on CGI, but I do know that gbrowse itself has its own timeout settings, which may be what you are running into.

In your main GBrowse.conf (usually in /etc/gbrowse2), there are these settings
slave_timeout          = 45
global_timeout         = 60
search_timeout         = 15

which I assume are seconds for the relative timeout. Unfortunately, no specific timeout for file uploads.

If you use fastcgi, there is another location in the gbrowse specific apache conf that controls various timeouts as well, including busy/idle or io.

So, yeah, I guess it may not be too surprising that the main process times out for the main display, but that the spawned process dealing with the file upload will continue in the background until finished, resulting in your wig file eventually showing up. I typically use fastcgi, with fastcgi timeouts of 3600 (the default?), and I do not see the timeouts when I upload large files.

I hope that helps,
Tim

On Feb 12, 2014, at 11:56 AM, Kevin Schaper <[hidden email]<mailto:[hidden email]>> wrote:

> Hi,
>
> While doing pre-release testing our new GBrowse2 instance, we found that .wig file uploads claim to fail...but then the track appears a short time later.
>
> We're currently just using regular cgi, I have the Timeout set at 1200.  (before I had set it, I would see timeout messages in the apache error log, though, it still showed the same error message while eventually succeeding)
>
> What I found was that this message seems to appear at 90 seconds when uploading from a file, and 60 seconds when uploading from a URL.  From the lines being printed on the screen, it doesn't seem to do it after the same amount of data has been transferred.  (~1M lines for url, ~1.3M lines for file) - so it seems more like a time thing than a size thing.
>
> Watching the post & response traffic with firebug, it doesn't look like anything comes from the server side to say that it's failing.  Does the front-end code have automatic timeouts to assume failure after a set amount of time?
>
> thanks,
> Kevin
>
>
>
>
>
> ------------------------------------------------------------------------------
> Android apps run on BlackBerry 10
> Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
> Now with support for Jelly Bean, Bluetooth, Mapview and more.
> Get your Android app in front of a whole new audience.  Start now.
> http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk_______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]<mailto:[hidden email]>
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse




------------------------------------------------------------------------------
Android apps run on BlackBerry 10
Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
Now with support for Jelly Bean, Bluetooth, Mapview and more.
Get your Android app in front of a whole new audience.  Start now.
http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse