Solved: Keeping state in Sentry onpremise with Kubernetes pod

On-Premise
1

Hi all, I have got Sentry onpremise running on a Kubernetes setup. Everything works fine so far but if I recreate the pod, the postgres database gets recreated and I have to re-run “sentry upgrade” on the web container. This means I lose all my previous setup and data in Sentry.

I have created a persistent volume and mounted it under /var/lib/postgresql on my postgres container. I have confirmed that if I create a file under this directory, it survives the deletion of the pod. But I still have to reinitialize Sentry.

My kubernetes deployment is configured as follows:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sentry
  labels:
    name: sentry
    env: staging
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: sentry
    spec:
      containers:
      - name: memcached
        image: memcached:1.4
      - name: redis
        image: redis:3.2-alpine
      - name: postgres
        image: postgres:9.5
        volumeMounts:
          - mountPath: "/var/lib/postgresql"
            name: postgres-data
      - name: smtp
        image: tianon/exim4
      - name: web
        image: repo/mycompany_sentry:0.1.1
        env:
        - name: SENTRY_USE_SSL
          value: '1'
        - name: SENTRY_SECRET_KEY
          value: ****
        - name: SENTRY_REDIS_HOST
          value: localhost
        - name: SENTRY_MEMCACHED_HOST
          value: localhost
        - name: SENTRY_POSTGRES_HOST
          value: localhost
        - name: SENTRY_EMAIL_HOST
          value: localhost
        - name: SENTRY_SERVER_EMAIL
          value: sentry@mycompany.com
      - name: worker
        image: repo/mycompany_sentry:0.1.1
        env:
        - name: SENTRY_SECRET_KEY
          value: ****
        - name: SENTRY_REDIS_HOST
          value: localhost
        - name: SENTRY_MEMCACHED_HOST
          value: localhost
        - name: SENTRY_POSTGRES_HOST
          value: localhost
        args:
        - run
        - worker
      - name: cron
        image: repo/mycompany_sentry:0.1.1
        volumeMounts:
        - name: secrets
          mountPath: /etc/nginx/conf.d/ssl
          readOnly: true
        env:
        - name: SENTRY_SECRET_KEY
          value: ****
        - name: SENTRY_REDIS_HOST
          value: localhost
        - name: SENTRY_MEMCACHED_HOST
          value: localhost
        - name: SENTRY_POSTGRES_HOST
          value: localhost
        args:
        - run
        - cron
      - name: nginx
        image: repo/sentry_nginx:0.1.2
        ports:
        - name: sentry-http
          containerPort: 80
        - name: sentry-https
          containerPort: 443
        volumeMounts:
        - name: secrets
          mountPath: /etc/nginx/conf.d/ssl
          readOnly: true
      volumes:
        - name: postgres-data
          persistentVolumeClaim:
            claimName: mycompany-sentry-postgres
        - name: secrets
          secret:
            secretName: mycompany-com-cert

Am I missing something? Do I need persistent volumes for something else, or is this expected behaviour?

Any help appreciated.

James

2

That is definitely not expected behavior, and I’m not familiar enough with Kubernetes to suggest anything else. But it definitely just sounds like the data for Postgres is not persisting between container restarts.

3

OK thanks Matt, I’ll see if I can replicate the issue locally in Docker without Kubernetes.

4

OK, I solved it. I had a look at the Dockerfile for the postgres image and found this:

VOLUME /var/lib/postgresql/data

So I modified my volume mount and set the data directory to a subdirectory:

    env:
    - name: PGDATA
      value: /var/lib/postgresql/data/pgdata
    volumeMounts:
      - mountPath: "/var/lib/postgresql/data"
        name: postgres-data

As recommended in this guide: https://blog.oestrich.org/2015/08/running-postgres-inside-kubernetes/

And now, if I delete and recreate the deployment, the data is not lost.

5

Hey I am traying to deploy sentry on kubernetes but I cant start web sentry, can you share with me your docker file for create your sentry image? and how do you create the admin user?