solr is broken...

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

solr is broken...

joe carlson
Hi Folks,

I have to admit I’ve been neglecting our intermine instance for the past couple months. Only today i was told that search was broken and I’m a bit flummoxed as to what is wrong.

It’s a standard setup of solr 7.2.1 and used to work. The one nonstandard issue is that we’re running everything in containers now. Hitting the /solr/ endpoint works, but /solr/schema/fields gives a 500 response. The error message I see on the console

> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
> WARNING: Unable to start the Restlet
> Internal Server Error (500) - No SolrRequestInfo in this Thread!
>         at org.apache.solr.rest.RestManager.getRestManager(RestManager.java:241)
>         at org.apache.solr.rest.SolrSchemaRestApi.createInboundRoot(SolrSchemaRestApi.java:71)
>         at org.restlet.Application.getInboundRoot(Application.java:272)
…< long stack trace here>
>         at java.lang.Thread.run(Thread.java:748)
>
> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
> WARNING: Unable to start the Restlet
> Nov 18, 2020 10:53:40 PM org.restlet.routing.Filter doHandle
> WARNING: The filter org.restlet.engine.application.RangeFilter@5045cb9d was executed without a next Restlet attached to it.


Now, first, what is a Restlet and why was it unable to start?
solr tries to grab as much memory it can for indexes. For me the start page shows that memory usage is 99.5% (250 out of 251Gb.) is this worrisome? The JVM memory is at 19%, though. Is the memory usage an issue?
Has anyone had a problem like this?

Thanks,

Joe
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: solr is broken...

Daniela Butano-2
Hi Joe,
not sure if it was a typo but when you hit the schema/fields endpoint
you need to specify the collection name
e.g. http://localhost:8983/solr/biotestmine-search/schema/fields 
otherwise it returns 500

Daniela


> Hi Folks,
>
> I have to admit I’ve been neglecting our intermine instance for the
> past couple months. Only today i was told that search was broken and
> I’m a bit flummoxed as to what is wrong.
>
> It’s a standard setup of solr 7.2.1 and used to work. The one
> nonstandard issue is that we’re running everything in containers now.
> Hitting the /solr/ endpoint works, but /solr/schema/fields gives a 500
> response. The error message I see on the console
>
>> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
>> WARNING: Unable to start the Restlet
>> Internal Server Error (500) - No SolrRequestInfo in this Thread!
>>         at
>> org.apache.solr.rest.RestManager.getRestManager(RestManager.java:241)
>>         at
>> org.apache.solr.rest.SolrSchemaRestApi.createInboundRoot(SolrSchemaRestApi.java:71)
>>         at
>> org.restlet.Application.getInboundRoot(Application.java:272)
> …< long stack trace here>
>>         at java.lang.Thread.run(Thread.java:748)
>>
>> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
>> WARNING: Unable to start the Restlet
>> Nov 18, 2020 10:53:40 PM org.restlet.routing.Filter doHandle
>> WARNING: The filter
>> org.restlet.engine.application.RangeFilter@5045cb9d was executed
>> without a next Restlet attached to it.
>
>
> Now, first, what is a Restlet and why was it unable to start?
> solr tries to grab as much memory it can for indexes. For me the start
> page shows that memory usage is 99.5% (250 out of 251Gb.) is this
> worrisome? The JVM memory is at 19%, though. Is the memory usage an
> issue?
> Has anyone had a problem like this?
>
> Thanks,
>
> Joe
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: solr is broken...

joe carlson
Hi Daniela,

Yes. That was a typo. Thanks.

I haven’t totally figured it out, but I think the issue is my solr db got corrupted or otherwise mucked with.

Coincidentally (or not), the issue seems to have arisen shortly after our local security scanners complained about my open port. I wouldn’t be surprised it the scanner modified some things.

joe

> On Nov 20, 2020, at 2:59 AM, [hidden email] wrote:
>
> Hi Joe,
> not sure if it was a typo but when you hit the schema/fields endpoint you need to specify the collection name
> e.g. http://localhost:8983/solr/biotestmine-search/schema/fields otherwise it returns 500
>
> Daniela
>
>
>> Hi Folks,
>> I have to admit I’ve been neglecting our intermine instance for the
>> past couple months. Only today i was told that search was broken and
>> I’m a bit flummoxed as to what is wrong.
>> It’s a standard setup of solr 7.2.1 and used to work. The one
>> nonstandard issue is that we’re running everything in containers now.
>> Hitting the /solr/ endpoint works, but /solr/schema/fields gives a 500
>> response. The error message I see on the console
>>> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
>>> WARNING: Unable to start the Restlet
>>> Internal Server Error (500) - No SolrRequestInfo in this Thread!
>>>        at org.apache.solr.rest.RestManager.getRestManager(RestManager.java:241)
>>>        at org.apache.solr.rest.SolrSchemaRestApi.createInboundRoot(SolrSchemaRestApi.java:71)
>>>        at org.restlet.Application.getInboundRoot(Application.java:272)
>> …< long stack trace here>
>>>        at java.lang.Thread.run(Thread.java:748)
>>> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
>>> WARNING: Unable to start the Restlet
>>> Nov 18, 2020 10:53:40 PM org.restlet.routing.Filter doHandle
>>> WARNING: The filter org.restlet.engine.application.RangeFilter@5045cb9d was executed without a next Restlet attached to it.
>> Now, first, what is a Restlet and why was it unable to start?
>> solr tries to grab as much memory it can for indexes. For me the start
>> page shows that memory usage is 99.5% (250 out of 251Gb.) is this
>> worrisome? The JVM memory is at 19%, though. Is the memory usage an
>> issue?
>> Has anyone had a problem like this?
>> Thanks,
>> Joe
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: solr is broken...

joe carlson
Rebuilding the solr index appears to solve the problem. But I’m still a bit perplexed by the stack trace. restlet (from what Prof. Google tells me) translates REST requests into solr requests. Not a very informative error message, in my opinion.

Joe

> On Nov 20, 2020, at 7:44 AM, Joe Carlson <[hidden email]> wrote:
>
> Hi Daniela,
>
> Yes. That was a typo. Thanks.
>
> I haven’t totally figured it out, but I think the issue is my solr db got corrupted or otherwise mucked with.
>
> Coincidentally (or not), the issue seems to have arisen shortly after our local security scanners complained about my open port. I wouldn’t be surprised it the scanner modified some things.
>
> joe
>
>> On Nov 20, 2020, at 2:59 AM, [hidden email] wrote:
>>
>> Hi Joe,
>> not sure if it was a typo but when you hit the schema/fields endpoint you need to specify the collection name
>> e.g. http://localhost:8983/solr/biotestmine-search/schema/fields otherwise it returns 500
>>
>> Daniela
>>
>>
>>> Hi Folks,
>>> I have to admit I’ve been neglecting our intermine instance for the
>>> past couple months. Only today i was told that search was broken and
>>> I’m a bit flummoxed as to what is wrong.
>>> It’s a standard setup of solr 7.2.1 and used to work. The one
>>> nonstandard issue is that we’re running everything in containers now.
>>> Hitting the /solr/ endpoint works, but /solr/schema/fields gives a 500
>>> response. The error message I see on the console
>>>> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
>>>> WARNING: Unable to start the Restlet
>>>> Internal Server Error (500) - No SolrRequestInfo in this Thread!
>>>>       at org.apache.solr.rest.RestManager.getRestManager(RestManager.java:241)
>>>>       at org.apache.solr.rest.SolrSchemaRestApi.createInboundRoot(SolrSchemaRestApi.java:71)
>>>>       at org.restlet.Application.getInboundRoot(Application.java:272)
>>> …< long stack trace here>
>>>>       at java.lang.Thread.run(Thread.java:748)
>>>> Nov 18, 2020 10:53:40 PM org.restlet.Restlet handle
>>>> WARNING: Unable to start the Restlet
>>>> Nov 18, 2020 10:53:40 PM org.restlet.routing.Filter doHandle
>>>> WARNING: The filter org.restlet.engine.application.RangeFilter@5045cb9d was executed without a next Restlet attached to it.
>>> Now, first, what is a Restlet and why was it unable to start?
>>> solr tries to grab as much memory it can for indexes. For me the start
>>> page shows that memory usage is 99.5% (250 out of 251Gb.) is this
>>> worrisome? The JVM memory is at 19%, though. Is the memory usage an
>>> issue?
>>> Has anyone had a problem like this?
>>> Thanks,
>>> Joe
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev