Slack integration with on premise sentry

Hello,

I am working on setting up slack alerts to our on premise sentry installation. We are on sentry version 21.7.0.
The documents I have followed while setting it up are

Also the redirect URL, the option load URL those we configure in the slack app should be accessible ?
I have configured as per the document above, but those URLS when used along with our sentry domain… they don’t work or exist!

While configuring alert rules, I could see that the drop down does show to select the slack-workspace option. But we are not receiving any alerts yet.
Even an email alert is not coming. So its either that my alert rules are not working or is there a global setting to enable or disable alert notifications ?

I don’t see any specific errors in the logs

13:38:00 [INFO] sentry.integrations.slack: slack.event.link_shared (slack_team_id='XXXXXX' slack_event_id='XXXXXX' slack_api_app_id='XXXXXX')
13:38:00 [INFO] sentry.integrations.slack: slack.link-shared (slack_shared_link='organizations/{organization}/alerts/metric-rules/project-information/2/')
13:11:55 [INFO] sentry.integrations.slack: slack.event.link_shared (slack_team_id='XXXXXXX' slack_event_id='XXXXXX' slack_api_app_id='XXXXXXX')
13:11:55 [INFO] sentry.integrations.slack: slack.link-shared (slack_shared_link='organizations/{organization}/alerts/metric-rules/project-information/2/')

Any help will be greatly appreciated.

Thanks

Can you expand on what you mean by this?

So as per the document, I have setup for example the

request URL as https://our-senrty-domain.envs.io/extensions/slack/action/
options load URL as https://our-senrty-domain.envs.io/extensions/slack/options-load/
redirect URL as https://our-senrty-domain.envs.io/extensions/slack/setup/

But when I access those URLS they don’t work or exist as its Page Not Found error. So I was wondering, if they are proper and need to be changed

Yeah that indicates a problem. It either means your Slack extension/plugin is not active or some other issue accessing your Sentry instance.

okay, I am sure as well that it indicates a problem :slight_smile:
Can you tell me, what has to be done other than the steps from the document https://develop.sentry.dev/integrations/slack/ that allows me to make use of slack alerts ?

It either means your Slack extension/plugin is not active
Also can you elaborate on on what you meant by my extension or plugin is not active ?

Those steps should suffice. The endpoints not being reachable seems like network configuration issue on your end. Possibly with your load balancer or reverse proxy?

Can you verify those requests make it to your Sentry web service at all?

Okay… I have checked the same. I could say all requests are hitting the nginx pod, but not the sentry pods with role as web.

While accessing the URLS, below listed are the response

The whole setup is installed in our Kubernetes cluster and the ingress is allowing requests to /

spec:
  rules:
  - host: senrty-domain.envs.io
    http:
      paths:
      - backend:
          service:
            name: sentry-nginx
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

Do you suggest any changes in the ingress rules ?

thanks!

I’d guess something funky going on with host-headers and maybe your system.url-prefix setting.

Yea… I agree to you, this value was not set specifically.

By setting this to our sentry domain name https://our-sentry-domain.envs.io/ I could see below errors in the sentry-web pod

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sentry/../sentry_sdk/transport.py", line 340, in send_envelope_wrapper
    self._send_envelope(envelope)
  File "/usr/local/lib/python3.6/site-packages/sentry/../sentry_sdk/transport.py", line 267, in _send_envelope
    endpoint_type="envelope",
  File "/usr/local/lib/python3.6/site-packages/sentry/../sentry_sdk/transport.py", line 180, in _send_request
    headers=headers,
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/request.py", line 72, in request
    **urlopen_kw)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/request.py", line 150, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/poolmanager.py", line 324, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 386, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='our-sentry-domain.envs.io', port=443): Read timed out. (read timeout=<object object at 0x7f6761e931e0>)
08:02:43 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

which can be expected, as the access of our sentry is authenticated via google oauth. Do you think, I can try setting it to the internal k8s service name ?

Is this Google Cloud Platform’s IAP? (identity aware proxy)

we are using google api credentials and setting up Oauth. I have also tried setting the system.url-prefix to http://sentry-nginx and still receives time out :roll_eyes:

May be, a change to be done in the nginx config ?

upstream relay {
  server sentry-relay:3000;
}

upstream sentry {
  server sentry-web:9000;
}

server {
  listen 8080;

  proxy_redirect off;
  proxy_set_header Host $host;

  location /api/store/ {
    proxy_pass http://relay;
  }

  location ~ ^/api/[1-9]\d*/ {
    proxy_pass http://relay;
  }

  location / {
    proxy_pass http://sentry;
  }
}

Also noticed that initially in the logs they throw a 502 bad gateway

10:14:59 [ERROR] sentry_sdk.errors: Unexpected status code: 502 (body: b'<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n')
Traceback (most recent call last):

after that followed by time outs.

I have made some progress. Its not a bad gateway any more, but sentry is forbidding the access considering CSRF

11:01:45 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/1/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/1/envelope/'>)
11:01:45 [ERROR] sentry_sdk.errors: Unexpected status code: 403 (body: b'\n\n\n\n\n\n\n\n\n\n<!DOCTYPE html>\n<html lang="en">\n<head>\n  <meta http-equiv="content-type" content="text/html; charset=utf-8">\n  <meta name="robots" content="NONE,NOARCHIVE">\n  <meta name="viewport" content="width=device-width, initial-scale=1">\n  <meta name="theme-color" content="#000000">\n\n  <link rel="icon" type="image/png" href="http://sentry-web:9000/_static/1629801987/sentry/images/favicon.png">\n\n  <link rel="apple-touch-icon" href="http://sentry-web:9000/_static/1629801987/sentry/images/logos/apple-touch-icon.png">\n  <link rel="apple-touch-icon" sizes="76x76" href="http://sentry-web:9000/_static/1629801987/sentry/images/logos/apple-touch-icon-76x76.png">\n  <link rel="apple-touch-icon" sizes="120x120" href="http://sentry-web:9000/_static/1629801987/sentry/images/logos/apple-touch-icon-120x120.png">\n  <link rel="apple-touch-icon" sizes="152x152" href="http://sentry-web:9000/_static/1629801987/sentry/images/logos/apple-touch-icon-152x152.png">\n\n  <link rel="mask-icon" sizes="any" href="http://sentry-web:9000/_static/1629801987/sentry/images/logos/logo-sentry.svg" color="#FB4226">\n\n  <link href="/_static/dist/sentry/entrypoints/sentry.css?v=6a3c273d2af7defa1580a16c6755f126abf8f5bf" rel="stylesheet"/>\n\n  \n\n  <title>CSRF Verification Failed | Sentry</title>\n\n  \n    <script>window.__initialData = {"singleOrganization":true,"supportEmail":"devops@company.com","urlPrefix":"http://sentry-web:9000","version":{"current":"21.7.0","latest":"21.8.0","build":"6a3c273d2af7defa1580a16c6755f126abf8f5bf","upgradeAvailable":true},"features":[],"distPrefix":"/_static/dist/sentry/","needsUpgrade":false,"dsn":"http://XXXXXXXX@sentry-web:9000/1","dsn_requests":"","statuspage":null,"messages":[],"apmSampling":0.0,"isOnPremise":true,"invitesEnabled":true,"gravatarBaseUrl":"https://secure.gravatar.com","termsUrl":null,"privacyUrl":null,"lastOrganization":null,"languageCode":"en","userIdentity":{"ip_address":"10.92.14.1"},"csrfCookieName":"sc","sentryConfig":{"dsn":"http://XXXXXXXX@sentry-web:9000/1","release":"6a3c273d2af7defa1580a16c6755f126abf8f5bf","environment":"production","whitelistUrls":[]},"demoMode":false,"isAuthenticated":false,"user":null};</script>\n  \n\n  <script>// if the ads.js file loads below it will mark this variable as false\n    window.adblockSuspected = true;\n    // Initialize this so that we can queue up tasks when Sentry SPA is initialized\n    window.__onSentryInit = window.__onSentryInit || [];</script>\n\n  <script>try {\n    var reg = new RegExp(/\\/organizations\\/(.+?(?=(\\/|$)))(\\/|$)/, \'i\');\n    var organization = window.location.pathname;\n    var slug = organization.match(reg)[1];\n\n    function promiseRequest(url) {\n      return new Promise(function (resolve, reject) {\n        var xhr = new XMLHttpRequest();\n        xhr.open(\'GET\', url);\n        xhr.onload = function () {\n          try {\n            this.status >= 200 && this.status < 300\n              ? resolve(JSON.parse(xhr.response))\n              : reject([this.status, this.statusText]);\n          } catch(e) {\n            reject();\n          }\n        };\n        xhr.onerror = function () {\n          reject([this.status, this.statusText]);\n        };\n        xhr.send();\n      });\n    }\n\n    function makeUrl(suffix) {\n      return \'/api/0/organizations/\' + slug + suffix;\n    }\n\n\n    // There are probably more, but this is at least one case where\n    // this should not be treated as a slug\n    if (slug !== \'new\') {\n      var preloadPromises = { orgSlug: slug };\n      window.__sentry_preload = preloadPromises;\n\n      preloadPromises[\'organization?detailed=0\'] = promiseRequest(makeUrl(\'/?detailed=0\'));\n      preloadPromises.projects =  promiseRequest(makeUrl(\'/projects/?all_projects=1&collapse=latestDeploys\'));\n      preloadPromises.teams = promiseRequest(makeUrl(\'/teams/\'));\n    }\n  } catch(_) {}</script>\n\n  \n  \n    \n    <script src="/_static/dist/sentry/entrypoints/app.js?v=6a3c273d2af7defa1580a16c6755f126abf8f5bf"></script>\n  \n\n  \n  \n\n  \n  <script src="/_static/1629801987/sentry/js/ads.js"></script>\n  \n</head>\n\n<body class=" narrow">\n  \n  <div class="app">\n    \n\n    \n    \n\n\n<div id="blk_alerts" class="messages-container"></div>\n<div id="blk_indicators"></div>\n\n<script>window.__onSentryInit = window.__onSentryInit || [];\n  window.__onSentryInit.push({\n    name: \'renderReact\',\n    component: \'SystemAlerts\',\n    container: \'#blk_alerts\',\n    props: {\n      className: \'alert-list\',\n    },\n  });\n  window.__onSentryInit.push({\n    name: \'renderReact \',\n    component: \'Indicators\',\n    container: \'#blk_indicators\',\n    props: {\n      className: \'indicators-container\',\n    },\n  });</script>\n\n\n\n    \n\n    <div class="container">\n      <div class="content">\n        \n<div class="pattern-bg"></div>\n<section class="org-login">\n  <div class="box box-modal">\n    <div class="box-header">\n      \n        \n          \n        \n        <a class="logo-with-action" href="/">\n          \n            <span class="icon-sentry-logo-full"></span>\n          \n          \n        </a>\n      \n    </div>\n    <div class="box-content with-padding">\n      \n    <section class="body">\n        <div class="page-header">\n            <h2>CSRF Verification Failed</h2>\n        </div>\n\n        <p>A required security token was not found or was invalid.</p>\n\n        <p>If you\'re continually seeing this issue, try the following:</p>\n\n        <ol>\n          <li>Clear cookies (at least for Sentry\'s domain).</li>\n          <li>Reload the page you\'re trying to submit (don\'t re-submit data).</li>\n          <li>Re-enter the information, and submit the form again.</li>\n        </ol>\n\n        \n\n        <p>Read more about <a href="http://en.wikipedia.org/wiki/Cross-site_request_forgery">CSRF on Wikipedia</a>.</p>\n    </section>\n\n    </div>\n  </div>\n</section>\n\n      </div>\n    </div>\n  </div>\n  \n\n  \n  \n  <script>document.addEventListener(\'DOMContentLoaded\', function() {\n      const el = document.querySelector(\'#sign-out\');\n      if (el) {\n        el.addEventListener(\'click\', function() {\n          document.modalLogoutForm.submit();\n        });\n      }\n    });</script>\n\n</body>\n</html>\n')

I have set the url-prefix value to our domain name and I don’t see any errors in the web logs. But we don’t receive any notifications :frowning:

This is because that request should go to relay not sentry web.

Were you able to finish Slack setup?

I do not see any errors as before, like time outs or bad gateways… but still alerts doesn’t appear in our slack channels.

do you have any other suggestions on troubleshooting this situation ? thanks

We have got the notifications to slack working.
But, it works for new issues created in sentry and doesn’t work for alert rules based on metrics.

So slack notification is working but not for alert rules based on metrics. Anyone has faced the same ?

I stumbled upon this post from you, does it mean its not supported ? :slight_smile:

Waiting for Fix for metrics alerts not firing by sagor999 · Pull Request #459 · sentry-kubernetes/charts · GitHub to be merged and I will give a try.