AnnotationChangeNotificationService Timeout Error

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

AnnotationChangeNotificationService Timeout Error

aaguilar
Hello Everyone,

I came across a problem that I hope someone can help me figure out.  I
installed webapollo on a dedicated server and one is able to connect to it
through a proxy.  Everything seemed to be working fine, but then the webserver
that is used as a proxy to webapollo crashed and we noticed that Apache had a
MaxClients error as well as thousands of Proxy timeout errors because it was
unable to contact WebApollo.  After some searching, we found out that many
Proxy timeout errors can cause the MaxClients.  From this, we concluded that
the webserver that we use to access webapollo crashed because of the thousands
of proxy errors the webserver has due to webapollo.

After some more investigating, we were able to figure out that the thing that
causes the Proxy timeout error is the AnnotationChangeNotificationService.  It
appears that when one is accessing WebApollo through a browser, an AJAX call
is done through the xhr.js file to the AnnotationChangeNotificationService.
Through the debugger, we were able to see that this AJAX call would always say
pending and then it would timeout.  Which also corresponded to the proxy
errors we saw in the Apache logs and the 504 error in the access log of
tomcat.  Do you guys know why I am experiencing this problem with this service
of WebApollo and how to fix it?

Sorry for the long explanation, just wanted to make sure I was well
understood :).  Hopefully my problem makes sense.

Thanks!
Reply | Threaded
Open this post in threaded view
|

RE: AnnotationChangeNotificationService Timeout Error

Antelmo Aguilar
Hello All,

I still have not been able to find a solution for the problem that I stated
below.  I would really appreciate it if anyone could give me an idea as to
what I might be able to do to fix this issue.  I will gladly provide any
information that might be needed.  I appreciate any help.

Thanks!

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of [hidden email] [[hidden email]]
Sent: Thursday, June 05, 2014 1:01 PM
To: [hidden email]
Subject: [apollo] AnnotationChangeNotificationService Timeout Error

Hello Everyone,

I came across a problem that I hope someone can help me figure out.  I
installed webapollo on a dedicated server and one is able to connect to it
through a proxy.  Everything seemed to be working fine, but then the webserver
that is used as a proxy to webapollo crashed and we noticed that Apache had a
MaxClients error as well as thousands of Proxy timeout errors because it was
unable to contact WebApollo.  After some searching, we found out that many
Proxy timeout errors can cause the MaxClients.  From this, we concluded that
the webserver that we use to access webapollo crashed because of the thousands
of proxy errors the webserver has due to webapollo.

After some more investigating, we were able to figure out that the thing that
causes the Proxy timeout error is the AnnotationChangeNotificationService.  It
appears that when one is accessing WebApollo through a browser, an AJAX call
is done through the xhr.js file to the AnnotationChangeNotificationService.
Through the debugger, we were able to see that this AJAX call would always say
pending and then it would timeout.  Which also corresponded to the proxy
errors we saw in the Apache logs and the 504 error in the access log of
tomcat.  Do you guys know why I am experiencing this problem with this service
of WebApollo and how to fix it?

Sorry for the long explanation, just wanted to make sure I was well
understood :).  Hopefully my problem makes sense.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: AnnotationChangeNotificationService Timeout Error

Colin
In reply to this post by aaguilar
Hi there,
This problem might be due to Comet "long polling" method that WebApollo uses. With this model, the server must keep connections open and listen for timeouts from the client, so the behavior you are seeing is sort of by design.
Some options you have though
1) You could try and set ProxyTimeout to a very large value to force connections to be left open (necessary for the long polling method!). I haven't tested this myself but I think it could help the situation. T
2) Another option is possibly using nginx for the proxy server, which is probably more scalable since it doesn't have the "one request per thread" model.
3) The final option that we are working towards is using websockets for WebApollo, which will alleviate these problems altogether, but it will require Tomcat 8 and is still in beta.

Let me know if there ProxyTimeout method helps. I may test this out myself since it seems important to get this working.
Thanks,
-Colin


On Thu, Jun 5, 2014 at 12:01 PM, <[hidden email]> wrote:
Hello Everyone,

I came across a problem that I hope someone can help me figure out.  I
installed webapollo on a dedicated server and one is able to connect to it
through a proxy.  Everything seemed to be working fine, but then the webserver
that is used as a proxy to webapollo crashed and we noticed that Apache had a
MaxClients error as well as thousands of Proxy timeout errors because it was
unable to contact WebApollo.  After some searching, we found out that many
Proxy timeout errors can cause the MaxClients.  From this, we concluded that
the webserver that we use to access webapollo crashed because of the thousands
of proxy errors the webserver has due to webapollo.

After some more investigating, we were able to figure out that the thing that
causes the Proxy timeout error is the AnnotationChangeNotificationService.  It
appears that when one is accessing WebApollo through a browser, an AJAX call
is done through the xhr.js file to the AnnotationChangeNotificationService.
Through the debugger, we were able to see that this AJAX call would always say
pending and then it would timeout.  Which also corresponded to the proxy
errors we saw in the Apache logs and the 504 error in the access log of
tomcat.  Do you guys know why I am experiencing this problem with this service
of WebApollo and how to fix it?

Sorry for the long explanation, just wanted to make sure I was well
understood :).  Hopefully my problem makes sense.

Thanks!