Unable to upgrade, stuck on version 9.1.2

On-Premise
21

@BYK
I have managed anyhow to login and noticed that all projects are in place, yet the events are missing.

Also after making the changes what would be right way to apply them?

22

Still nothing from https://sentry.uhlhost.net/

23

I have noticed that the local service from ubuntu is masked.

Also these are the only places where my custom nginx proxy config is mentioned. It cannot work if the nginx was replaced by docker and docker does not have the right config in place. Seems there are some changes at nginx integration in the newer releases of sentry.

This was working great with the old config and nginx was serving via https from web:9000

**cosmic@sentry** : **/etc/nginx/sites-available** $ sudo locate sentry_uhlhost_net.conf
    /etc/nginx/sites-available/sentry_uhlhost_net.conf
    /etc/nginx/sites-enabled/sentry_uhlhost_net.conf
    /var/snap/docker/common/var-lib-docker/aufs/diff/1ea4827932b585c13af1fe318ac30b25d4d86f0b5ecab7f4c69e1ba8876a3f81/etc/nginx/sites-enabled/sentry_uhlhost_net.conf
24

When you say load balancer you reffer to local nginx deployment. What is the role of the docker nginx? This wasnt in place when I used it with version 9.1.2, local version of nginx was running, now the service is masked as seen above.

25

I mean anything in front of the docker setup we provide, that would do the SSL termination and potentially other stuff. The built-in nginx is for routing requests between the Sentry Web service and the new Relay service, which is the service that ingests and processes the events now.

26

@BYK
Well like I mentioned in the above replies, I have done that you described and it did not helped. Nginx was routing it all well before the upgrade, and we had SSL working. Now I cant seem to find out what is blocking it why is not working.

Please advise website is unusable uder https il have to chage all code and add insecure host with :9000 thats not the desired state of the app to have in production.

And we did not expect the upgrade will kill our current nginx connection.

We followed in past the docs we found to add nginx as a proxy for sentry, was working for years, up until now.

27

This is not expected from our side either. There is nothing in the setup that would block requests coming in from a another reverse proxy. I recommend you checking your nginx logs on both instances to debug the issue.

28

I have the old sentry backed up, only thing I can say is that our

docker-compose.yml old config used to have and worked with local nginx proxy:

services:
  nginx:
      image: nginx:latest
      restart: unless-stopped
      hostname: nginx
      ports:
         - "80:80"
         - "25:25"
         - "465:465"
         - "443:443"
         - "587:587"
      links:
        - web
      depends_on:
        - web
      volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/nginx/certs:/etc/nginx/certs
      - /etc/nginx/sites-enabled/sentry_uhlhost_net.conf:/etc/nginx/sites-enabled/sentry_uhlhost_net.conf

While the new docker-compose.yml has and does not work with our local nginx proxy:

nginx:
    << : *restart_policy
    ports:
      - '9000:80/tcp'
    image: "nginx:1.16"
    volumes:
      - type: bind
        read_only: true
        source: ./nginx
        target: /etc/nginx
    depends_on:
      - web
      - relay

Then this is completely new:

---
relay:
  upstream: "http://web:9000/"
  host: 0.0.0.0
  port: 3000
logging:
 level: WARN
processing:
  enabled: true
  kafka_config:
    - {name: "bootstrap.servers", value: "kafka:9092"}
  redis: redis://redis:6379
29

And yes currently nginx is disabled and masked the local deployment not the docker image.

I still do not get it why we need 3 http servers to take a website/app online.

  1. uWSGI - updated
  2. docker nginx
  3. local nginx

Something in between is terrible wrong, this is 2nd day no luck to track the issue.

30

@BYK
Can you point me to documentation for nginx as a proxy for sentry? For version 10 and above.

31

Hi, this seems pretty condescending, but I’m going to answer in the hope that it wasn’t intended to be that way.

uSWGI

It’s uWSGI, and this is a python web server. It’s needed to serve the Python application. Without this, the python application can’t run. You need some WSGI server.

docker nginx

This is needed because there are multiple services running behind our single hostname now. Specifically requests need to be routed between relay and the python web app. We include this so you don’t need to think about this and it works out of the box. As we change routes to go to Relay vs the Python app, etc, this routing is maintained by us so it doesn’t break every release.

local nginx

This seems to be your choice if you wanna terminate TLS or do anything else. You should think of Sentry’s docker deployment more as a black box. We’re attempting to provide a solution that just works and you as a user don’t need to be as aware of what moving pieces there are. So with that, if you attempt to ignore that we’re also running an nginx, just direct your nginx at port 9000 that nginx is bound to, and go from there. This should be opaque to you.

1 Like
32

@matt
Well no one tries to ignore anything, is just that in this mad loop from the upgrade of 9.1.2 to 20 we lost our local nginx and found it masked service. Not sure what caused that since we had it running perfectly service https://sentry.uhlhost.net/ its clearly now that it works over docker nginx and on port:9000 without ssl it breaks all our integrations.

Not to mention the missing events after the migration.

Still after few hours spent to track what blocked local nginx and disabled it completely i wasnt able to do so.

Tried to follow @BYK guide to enable SSL, yet this also did not helped while without any documentation in place it seems things changed in docker-compose.yml under nginx too, relay is new and wasnt present in older versions, all these somehow undocumented or maybe i miss that wiki or docs link, thats why I kindly asked if someone can point me to the docs, il be more than happy to recheck all files accordingly.

I had also with first deployment a strugle a bit to make nginx server as a proxy yet we managed it, now the upgrade broke it and we are left in this situation.

33

There are not one single issue on docker nginx, and the local nginx has not one single line of logs.

34

Seems for some reassons nginx was uninstalled completely, i have reinstalled it and now website can resolve over https://

@BYK - The only remaining issue would be to get all old events we have back in sentry. Any idea where I can start with this?

35

Alright now this is something that was not present before, and thus no idea what is causing this to be like this:

[zimbra@mail conf] host mail.uhlhost.net mail.uhlhost.net has address 137.74.247.200 [zimbra@mail conf] host sentry.uhlhost.net
sentry.uhlhost.net has address 92.222.242.243
If I go https://mail.uhlhost.net I get redirected to sentry… this wasnt happening before its first time when even such a redirect is visible to me. While always on mail.uhlhost.net I get zimbra mail client.
Please anyone advise how is this possible that they are hosted on different subnets and different machines.

The redirection got fixed not sure what caused it. It worked well on after only few minutes. I did a reboot on both machines.

36

It works fine from my end. I dunno what you’re doing man, but you seem to be deep in stuff we can’t help you with. This is well beyond the scope of Sentry at this point, and as a reminder, you’re using free software and being supported here for free. It seems like you need to learn how to debug your setups and debug your software. It’s pretty easy in general to debug how redirects work or how your nginx’s are configured. We’re not here to hold your hand through your own education.

2 Likes
37

@matt solved all issues mentioned above.

Managed to have sentry upgraded from 9.1.2 to latest version it works over nginx proxy serving with ssl.

Now like the one thing no one seems to notice is that all my events are gone. If anyone can help with this would be great.

@BYK would be great if now we could see why the old events are not moved and why no new events are incomming.

38

@UHLHosting - your logs suggest your Kafka instance is having a hard time staying up, which explains missing events and not ingesting any new events. You may be needing more memory on your server.

Also after all these exchanges, I sincerely feel like it would be easier and cheaper for you to use sentry.io instead of trying to maintain your own instance, unless you have other reasons to avoid our SaaS service.

39

@BYK

Right now we have 4 gb attached to instance. We could increase it if thats the case. Would like to keep our stack on our infra for now. Thank you.

40

@UHLHosting

Part of what you get from using sentry.io (SaaS) is the maintenance and support of using the product. We do not offer free support for on-premise; this thread has already gone above how we’d typically assist people.

I’d recommend increasing the RAM on the machine as my final response to this topic.