Setup with k8s no web access

On-Premise
1

Hi all. I’m trying to setup sentry on my baremetal k8s cluster. I seem to have hit a wall in trying to get the web interface to show.

Here is my YAML for deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sentry
  labels:
    name: sentry
    env: staging
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: sentry
    spec:
      containers:
      - name: redis
       image: redis:3.2-alpine
      - name: sentry
        image: 123.127.154:5000/sentry:latest
        env:
        - name: SENTRY_REDIS_HOST
          value: localhost
        - name: SENTRY_SECRET_KEY
          value: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
        - name: SENTRY_EMAIL_HOST
          value: mail.mydomain.com
        - name: SENTRY_EMAIL_PORT
          value: "587"
        - name: SENTRY_EMAIL_USER
          value: admin
        - name: SENTRY_EMAIL_PASSWORD
          value: XXXXXXXXXXXXXXX
        - name: SENTRY_EMAIL_USE_TLS
          value: "1"
        - name: SENTRY_ENABLE_EMAIL_REPLIES
          value: "1"
        - name: SENTRY_SMTP_HOSTNAME
          value: mail.mydomain.com
      - name: cron
        image: 123.127.154.11:5000/sentry:latest
        env:
        - name: SENTRY_REDIS_HOST
          value: localhost
        - name: SENTRY_SECRET_KEY
          value: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
        args:
        - run
        - cron
      - name: postgres
        image: postgres:9.5
        volumeMounts:
          - mountPath: "/var/lib/postgresql/data"
            name: postgres-data
      volumes:
         - name: postgres-data
           hostPath:
             path: /storage/gv0/postgres

kind: Service
apiVersion: v1
metadata:
  name: "sentry"
spec:
  selector:
    app: "sentry"
  ports:
    - name: "sentry"
      protocol: "TCP"
      port: 9000
      targetPort: 9000

It deploys successfully, but if I use an ssh tunnel for the pod’s IP on port 9000 it will not connect. I do see a path of auth/login, but it will timeout. If I run curl directly inside the sentry container it will give back an empty response. That’s not normal, right?

What am I missing here?

[root@master1 image-build]# kubectl exec -ti sentry-723606023-d9g22 -c sentry bash
root@sentry-723606023-d9g22:/usr/src/sentry# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
sentry       1  0.0  0.0   4228   680 ?        Ss   22:04   0:00 tini -- sentry run web
sentry      15  0.1  0.0  50504  9592 ?        S    22:04   0:02 [Sentry] uWSGI master
sentry      20  0.1  0.2 503180 82476 ?        Sl   22:04   0:02 [Sentry] uWSGI worker 1
sentry      21  0.1  0.2 513456 96460 ?        Sl   22:04   0:03 [Sentry] uWSGI worker 2
sentry      22  0.1  0.2 518928 96564 ?        Sl   22:04   0:03 [Sentry] uWSGI worker 3
root       135  0.0  0.0  21996  3652 ?        Ss   22:42   0:00 bash
root       141  0.0  0.0  19192  2412 ?        R+   22:42   0:00 ps aux

& in the cron container:

root@sentry-723606023-d9g22:/usr/src/sentry# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
sentry       1  0.0  0.0   4228   684 ?        Ss   22:04   0:00 tini -- sentry run cron
sentry      15  0.2  0.2 298452 88632 ?        S    22:04   0:05 [celery beat] run cron                              
root        20  0.0  0.0  21996  3684 ?        Ss   22:43   0:00 bash
root        27  0.0  0.0  19192  2412 ?        R+   22:44   0:00 ps aux

If I telnet the service addy on port 9000 from a host it immediately disconnects.

2

I obviously don’t really know what you’re doing, but at a cursory level, it looks like you have lots of things misconfigured. It seems that you’re trying to tell Sentry to talk to Redis on localhost, which obviously won’t be true since it’s in another container. It also seems that you’re missing Postgres. I’d suggest reading through instructions again.

3

Yes, it will be on localhost :slight_smile: This is kubernetes. They share the same pod. I’m not missing postgres, it’s right there! Just didn’t bother to show you the container’s PS AUX (it’s irrelevant)

[root@master1 ~]# kubectl exec -ti sentry-3403148151-xw9iu -c postgres sh
# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres     1  0.0  0.0 227204 20396 ?        Ss   23:37   0:00 postgres
postgres    25  0.0  0.0 227204  4172 ?        Ss   23:37   0:00 postgres: checkpointer process  
postgres    26  0.0  0.0 227204  5636 ?        Ss   23:37   0:00 postgres: writer process  
postgres    27  0.0  0.0 227204  4172 ?        Ss   23:37   0:00 postgres: wal writer process  
postgres    28  0.0  0.0 227624  6508 ?        Ss   23:37   0:00 postgres: autovacuum launcher process  
postgres    29  0.0  0.0  82188  4384 ?        Ss   23:37   0:00 postgres: stats collector process  
postgres    31  0.0  0.0 228128  7552 ?        Ss   23:38   0:00 postgres: postgres postgres 127.0.0.1(46278) idle
postgres    32  0.0  0.0 228128  9628 ?        Ss   23:38   0:00 postgres: postgres postgres 127.0.0.1(46282) idle
postgres    33  0.0  0.0 228128  9628 ?        Ss   23:38   0:00 postgres: postgres postgres 127.0.0.1(46280) idle
root        40  0.0  0.0   4336   800 ?        Ss   23:44   0:00 sh
root        46  0.0  0.0  19100  2436 ?        R+   23:44   0:00 ps aux
4

I’m not sure then. From this, I can only guess that it’s hanging from networking issues. Sockets not able to talk to their backends. You can debug this with strace to see where things are hanging, but other than that, there’s not much other insight I can give.

1 Like
5

Strace is a good idea, I’ll get on that. It does appear there is a redirect attempting to take place as well. I wonder if it’s trying to establish an https connection and erroring out without a cert. It would be nice if there was logs, I did take a look around var/logs but saw nothing relevant.

I’ll go ahead and probe around the processes directly with strace to see what’s going on.

Thanks.

Is this normal?

[root@master1 image-build]# kubectl logs sentry-2887003527-3vcsm sentry
*** Starting uWSGI 2.0.15 (64bit) on [Tue May 30 02:20:17 2017] ***
compiled with version: 4.9.2 on 13 May 2017 00:32:40
os: Linux-4.9.0-1.el7.elrepo.x86_64 #1 SMP Sun Dec 11 15:43:54 EST 2016
nodename: sentry-2887003527-3vcsm
machine: x86_64
clock source: unix
detected number of CPU cores: 48
current working directory: /usr/src/sentry
detected binary path: /usr/local/bin/uwsgi
!! no internal routing support, rebuild with pcre support !!
your memory page size is 4096 bytes
detected max file descriptor number: 65536
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to TCP address 0.0.0.0:9000 fd 4
Python version: 2.7.13 (default, May 11 2017, 21:39:56)  [GCC 4.9.2]
Set PythonHome to /usr/local
Python main interpreter initialized at 0x1160ba0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
setting request body buffering size to 65536 bytes
mapped 1922048 bytes (1877 KB) for 12 cores
*** Operational MODE: preforking+threaded ***
spawned uWSGI master process (pid: 15)
spawned uWSGI worker 1 (pid: 20, cores: 4)
spawned uWSGI worker 2 (pid: 21, cores: 4)
spawned uWSGI worker 3 (pid: 22, cores: 4)
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1160ba0 pid: 20 (default app)
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1160ba0 pid: 21 (default app)
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1160ba0 pid: 22 (default app)
[root@master1 image-build]# curl sentry.default.svc.kubernetes.local:9000
curl: (56) Recv failure: Connection reset by peer
6

That in particular sticks out. Does this always come up?

Appears to be non-issue.

stracing I go…

This repeats on the sentry run web process:

[root@master1 ~]# kubectl exec -ti sentry-2887003527-3vcsm -c sentry bash
root@sentry-2887003527-3vcsm:/usr/src/sentry# strace -p 1
Process 1 attached
wait4(-1, 0x7ffc089fc35c, WNOHANG, NULL) = 0
rt_sigtimedwait(~[ILL TRAP ABRT BUS FPE SEGV TTIN TTOU SYS RTMIN RT_1], 0x7ffc089fc2f0, {1, 0}, 8) = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7ffc089fc35c, WNOHANG, NULL) = 0
rt_sigtimedwait(~[ILL TRAP ABRT BUS FPE SEGV TTIN TTOU SYS RTMIN RT_1], 0x7ffc089fc2f0, {1, 0}, 8) = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7ffc089fc35c, WNOHANG, NULL) = 0

& this repeats on the master worker process:

root@sentry-2887003527-3vcsm:/usr/src/sentry# strace -p 15
Process 15 attached
getsockopt(4, SOL_TCP, TCP_INFO, "\n\0\0\0\0\0\0\0@B\17\0\0\0\0\0\30\2\0\0\0\0\0\0\0\0\0\0d\0\0\0"..., [104]) = 0
wait4(-1, 0x7ffcf82ecfe8, WNOHANG, NULL) = 0
epoll_wait(13, {}, 1, 1000)             = 0
getsockopt(4, SOL_TCP, TCP_INFO, "\n\0\0\0\0\0\0\0@B\17\0\0\0\0\0\30\2\0\0\0\0\0\0\0\0\0\0d\0\0\0"..., [104]) = 0
wait4(-1, 0x7ffcf82ecfe8, WNOHANG, NULL) = 0
epoll_wait(13, {}, 1, 1000)             = 0

In order or no?

7

Honestly, this is going to be really hard for me to debug your setup through forums by reading your strace logs. I was mostly suggesting you’d be able to figure it out by doing that. I can tell you that you’d need a -f in there though to get the children processes, not just the parent.

1 Like
8

No worries, think I’m gonna ditch the prebuilt docker image, build from source, and see if I can figure it out step by step. Since this service will remain private I can run it on a custom systemd image and build it all within a single container. That should simplify things significantly.

Appreciate the replies.

:smiley: :+1: