Skip to content

Logging

Dann edited this page Oct 19, 2020 · 3 revisions

ScholarSphere logs to splunk, below are some useful queries to find what you're looking for

Base queries

Non-Prod

index=libraries  | spath "k8s-cluster" | search "k8s-cluster"="psul-dev" | spath "kubernetes.namespace_name" | search "kubernetes.namespace_name"="scholarsphere-qa"
  • where "scholarsphere-qa" is the release name. if you pushed up "preview/new-font" the name would be "scholarsphere-new-font"

Production

index=libraries  | spath "k8s-cluster" | search "k8s-cluster"="psul-prod" | spath "kubernetes.namespace_name" | search "kubernetes.namespace_name"="scholarsphere"

Finding Service logs.

You can drill down into any service, or object you want.

<base-query> | search "kubernetes.container_name"="scholarsphere"

where "kubernetes.containername" is one of

scholarsphere # rails app
scholarsphere-sidekiq # sidekiq
redis
solr

To see all available service containers in the time period you selected:

<base-query> | stats count by kubernetes.container_name

Scenario

A user submits a ticket and says a work is giving them a 404 error, and they believe it should be.

  1. start with the base query
Screen Shot 2020-10-19 at 1 13 48 PM
  1. filter down to only the rails pods

drill_down

  1. Choose status 404 szoter_annotated_image(2)

In this example we see the user tried to get to the 1eb62dbd-9c15-49e8-bbdd-cd7535de8dcs0 resource. we look in the rails console and don't find that resource, but we do find a 1eb62dbd-9c15-49e8-bbdd-cd7535de8dcs resource, so this must be a typo on the users part, we reply to the message with the proper URL and the guest is super happy

Tips

  1. filtering IN is more performant than filtering OUT.
  2. cast a wide net, then filter down. i.e if you were looking for status code of 404, you would start the search by finding the rails container, and then searching for status == 404, and not the other way around.
  3. select a small time period, and select it first
Clone this wiki locally