Issues incorrectly re-opened as "regressed"


#1

My team recently started using Sentry to identify and detect and diagnose client-side issues in our app. We’re using the “releases” feature to correlate issues with specific releases and get source maps in Sentry for those issues. Loving it so far!

The only issue we’ve had is that after marking an issue as resolved in Sentry (and noting the version in which it was resolved), we’ll see a trickle of events roll in for the issue even after the fix has been deployed to production that cause the issue to be re-opened as a regression. These events are marked with newer release tags in which we know the issue no longer exists. Now, I realize that there’s a chance that we maybe didn’t fix the issue like we initially thought (i.e., it’s a true regression), but this has happened enough times now that I don’t think it’s just a mess up on our part. Also, the events do eventually stop coming in for the issue and it stays as “resolved” - it’s just annoying for developers to expect and issue to be re-opened several times before it’ll finally stay marked as “resolved”.

Has anyone else seen an issue like this where events continue to roll in after an issue has been resolved, where those events are tagged with a release in which you know the issue has been resolved?


#2

Are you pushing releases via the API? It’s possible that a release isn’t created in the system in the correct order.

e.g.

  1. you deploy v1
  2. you deploy v2
  3. an error happens in v2 (release gets created due to error)
  4. you mark issue as resolved in v3
  5. you deploy v3
  6. an error happens in v1 (release gets created)
  7. the error that you marked as resolved occurs in v1
  8. v1 marked as regression as the release has a newer ‘date created’ timestamp

#3

We’re creating releases in Sentry via a CI/CD Jenkins job, so they should all be created in order. Furthermore, the events that are causing the issues to be re-opened are all tagged with newer releases as indicated by the release versions themselves that are tagged onto the events.

The puzzling part is that these newer release versions all contain the same fix.