Identify real errors from errors generated by smoke tests

1

Hi,

After deploying my app, I run a set of independent smoke tests to ensure the app has been deployed correctly.

One of the tests is ensuring I cannot access a resource without passing auth creds. This returns a 401. The event is then sent to sentry.

I’d consider this white noise and would prefer for it isolated from real 401 errors.

Any ideas on how I can achieve this?

2

Hey @drvalani, you can check the IP address of the request (presumably this will be one from your internal network?) and set a different fingerprint if you identify the request to come from a smoke test. This will create a separate issue for smoke-test related errors which you can mute.

How this works in detail depends on which SDK you use.

3

You also might look into setting a tag when smoke tests are running.

4

Thanks for the quick replies @untitaker and @zeeg.

I like the idea of creating separate issues for smoke-test related errors. Is it possible to differentiate by passing say a value in the HEADER (ie, x-source: “smoke-test”) value rather than on IP?

Also @untitaker what does this mean “set a different fingerprint”.

@zeeg Smoke Tests can run at the same time as when real traffic is coming in, are you saying just to capture all events as smoke tests during this period?

Also I am using the python flask sdk.

5

I was thinking of this: https://docs.sentry.io/error-reporting/configuration/filtering/?platform=python#before-send

before_send allows you to modify the event shortly before it is sent to sentry. Fingerprints are a way to fix how events are grouped into issues if the default behavior is undesirable.

Something like this should work for Flask:

from flask import request

def is_a_smoke_test(request):
    # check headers, IP address, whatever

def before_send(event, hint):
    if request and is_a_smoke_test(request):
        # remove {{default}} if you want to group all smoke test
        # errors together (regardless of exception type)
        event['fingerprint'] = ["{{default}}", "smoke-test"]
        # return None  # uncomment to just drop the event
    return event

sentry_sdk.init(before_send=before_send)

You can also add tags or change the environment the same way

6

Yeah I’d probably suggest doing an environment or tag over fingerprint, otherwise you’re going to have some weird behavior in Sentry. A failing smoke test - after all - should be on the same degree as a real failure.

7

@zeeg I think the issue is that the smoke test is not actually failing, but rather they have events for each 401 which are asserted via the smoke test.

8

Ahhh! Maybe a separate project (and DSN) then?

9

I think what @untitaker has suggested will work.

We have our own build failure alerts for smoke test failure, so just silencing any noise from smoke is all we are really after.

Thanks for your help @zeeg and @untitaker