Relay missing key id project

mj.taleghani · February 23, 2021, 7:37am

hi there, recently I have set up a server configured to run sentry. I wrote all docker-compose files configs on my own and it is working for some projects but for a specific project I got bellow error from relay server:
relay | 2021-02-23T07:18:24Z [relay_server::actors::events] ERROR: project state for key ########### is missing key id
relay | 2021-02-23T07:18:25Z [relay_server::actors::events] ERROR: project state for key ########### is missing key id

can anyone help me to work around this problem?

BYK · March 4, 2021, 6:31pm

Hi there! Sorry but it is very hard for us to support custom setups as it is impossible to know all the details. Any reason you are not using our supported onpremise repo?

mj.taleghani · May 2, 2021, 8:41am

yes indeed.
your repo is just for those who completely access the internet. our servers are offline. I have to have internet access without any restrictions to docker repositories or any other sites that the on-premise rep needs to startup while running. Here I have restrictions to see docker repos or other areas.

BYK · May 3, 2021, 8:54am

It should be possible to download and ship those docker images to your offline network just like you do for our other code?

mj.taleghani · June 28, 2021, 5:55am

otherwise in any other questions which I saw here in the forum, even anyone asked about a problem with your repo, all answers he/she got is just we are thinking about, or ‘update the repo then the problem will solve!’.
these are not supported which need.
I rarely saw a problem has solved!
you can find some here:

mj.taleghani · June 28, 2021, 5:56am

On the other pages i neither see any helpful answers from you even when the requester are using your mentioned repository.

mj.taleghani · June 28, 2021, 5:58am

Or here too

github.com/getsentry/sentry

Error events are accepted

opened 10:47PM - 17 Dec 20 UTC

closed 11:12AM - 07 Jun 21 UTC

Fgruntjes

Component: Event Pipeline Status: Backlog

## Important Details How are you running Sentry? Using the helm chart https:…//github.com/sentry-kubernetes/charts running version Sentry 20.12.1bade073 with values ``` web: hpa: enabled: false cputhreshold: 60 minpods: 1 maxpods: 10 replicacount: 1 resources: requests: cpu: 300m memory: 500Mi env: - name: SENTRY_SLACK_CLIENT_ID value: "XXXXXXX" - name: SENTRY_SLACK_CLIENT_SECRET value: "XXXXXXX" - name: SENTRY_SLACK_VERIFICATION_TOKEN value: "XXXXXXX" - name: SENTRY_VSTS_CLIENT_ID value: "XXXXXXX" - name: SENTRY_VSTS_CLIENT_SECRET value: "XXXXXXX" cron: replicacount: 1 resources: requests: cpu: 150m memory: 200Mi env: - name: SENTRY_SLACK_CLIENT_ID value: "XXXXXXX" - name: SENTRY_SLACK_CLIENT_SECRET value: "XXXXXXX" - name: SENTRY_SLACK_VERIFICATION_TOKEN value: "XXXXXXX" - name: SENTRY_VSTS_CLIENT_ID value: "XXXXXXX" - name: SENTRY_VSTS_CLIENT_SECRET value: "XXXXXXX" worker: replicacount: 1 resources: requests: cpu: 150m memory: 200Mi env: - name: SENTRY_SLACK_CLIENT_ID value: "XXXXXXX" - name: SENTRY_SLACK_CLIENT_SECRET value: "XXXXXXX" - name: SENTRY_SLACK_VERIFICATION_TOKEN value: "XXXXXXX" - name: SENTRY_VSTS_CLIENT_ID value: "XXXXXXX" - name: SENTRY_VSTS_CLIENT_SECRET value: "XXXXXXX" system: url: "XXXXXXX" adminEmail: "XXXXXXX" secretKey: "XXXXXXX" mail: backend: smtp from: XXXXXXX host: XXXXXXX port: 25 useTls: true username: sentry password: XXXXXXX postgresql: enabled: true persistence: size: 30Gi kafka: enabled: true filestore: backend: s3 s3: accessKey: "XXXXXXX" secretKey: "XXXXXXX" bucketName: "XXXXXXX" endpointUrl: "XXXXXXX" region_name: "XXXXXXX" sentry: features: orgSubdomains: true service: type: ClusterIP nginx: enabled: false ingress: enabled: true hostname: XXXXXXX tls: - hosts: - XXXXXXX secretName: sentry-ingress-cert annotations: kubernetes.io/ingress.class: nginx certmanager.k8s.io/cluster-issuer: letsencrypt nginx.ingress.kubernetes.io/use-regex: "true" ``` ## Description Events are being accepted with an incorrect DSN ## Steps to Reproduce - Create a project and note the ID - Send curl request to push an incorrect sentry event: ``` curl --data '{"error": "someerror"}' \ -H 'Content-Type: application/json' \ -H "X-Sentry-Auth: Sentry sentry_version=7, sentry_key=3f642033bc56406d838f8a1d4a7321af, sentry_client=raven-bash/0.1" \ https://XXXX/api/9/store/ ``` I do see these logs in the `sentry-relay` pod: ``` 2020-12-17T22:40:53Z [relay_server::actors::events] ERROR: project state for key 43f7e281d3094142af2c7b7522673de5 is missing key id 2020-12-17T22:41:18Z [relay_server::actors::events] ERROR: project state for key 43f7e281d3094142af2c7b7522673de5 is missing key id 2020-12-17T22:41:35Z [relay_server::actors::events] ERROR: project state for key e066ddcb8eb24132856691078a934fd8 is missing key id 2020-12-17T22:41:48Z [relay_server::actors::events] ERROR: project state for key 43f7e281d3094142af2c7b7522673de5 is missing key id ``` ### What you expected to happen The event should be dropped ### Possible Solution Because most sentry instances running in K8S are not accessible from the outside I expect we just configured something wrong and there is some kind of settings in Sentry to force these keys. Any help will be greatly appreciated!

BYK · June 28, 2021, 8:15am

It is not possible to help you when you just share a single line of generic error message, sorry.

If you can provide more details we’d try to help. I don’t think digging the forum for posts to show us we are not being helpful is a good use of neither your nor our time.

mj.taleghani · July 4, 2021, 3:37am

how can i share info with you?
I have these line related ro relay server:

relay                                       |  ERROR relay_server::actors::events              > project state for key f2fc60f1d7d04ade83d98d38fe988429 is missing key id
relay                                       |  ERROR relay_server::actors::events              > project state for key f2fc60f1d7d04ade83d98d38fe988429 is missing key id
relay                                       |  DEBUG relay_server::actors::project             > project 48459058a971423a9e59584b05348cfd state requested
relay                                       |  DEBUG relay_server::actors::project             > project state 48459058a971423a9e59584b05348cfd updated
relay                                       |  ERROR relay_server::actors::events              > project state for key 48459058a971423a9e59584b05348cfd is missing key id

and relay service setting:

---
relay:
  mode: proxy
  #mode: static
  upstream: "http://sentry:9000/"
  tls_port: ~
  tls_identity_path: ~
  tls_identity_password: ~
logging:
  level: debug
  format: pretty
  enable_backtraces: true
processing:
  enabled: true
  kafka_config:
    - {name: "bootstrap.servers", value: "kafka:9092"}
    - {name: "message.max.bytes", value: 50000000} #50MB or bust
  redis: redis://sentry-redis:6379
  #geoip_path: "/geoip/GeoLite2-City.mmdb"
http:
  _client: "reqwest"

And my docker-compose.yml file:

version: '3'
#--------------------------------------------------------------
volumes:
  nginx_dir:
  nginx_logs:
  sentry_data:
  sentry_setting:
  sentry_config:
  pgdb_data:
  redis_data:
  relay_config:
    driver: "local"
  relay_geoip:
    driver: "local"
  clickhouse_config:
  clickhouse_log:
  clickhouse:
  sentry_symbolicator:
  zookeeper:
  zookeeper_log:
  zookeeper_secrets:
  kafka:
  kafka_log:
  kafka_settings:
#--------------------------------------------------------------
services:
  nginx:
    image: nginx:1.19.6
    container_name: nginx
    restart: always
    environment:
      TZ: "Asia/Tehran"
    depends_on:
      - "relay"
    volumes:
      - "nginx_dir:/etc/nginx"
      - "nginx_logs:/var/log/nginx"
    ports:
      - "80:80"
      - "443:443"
#--------------------------------------------------------------
  sentry-redis:
    image: redis:5
    container_name: sentry-redis
    restart: always
    privileged: true
#    environment:
#      TZ: "Asia/Tehran"
    volumes:
      - "redis_data:/data"
#--------------------------------------------------------------
  sentry-postgres:
    image: postgres:9.6.20
    container_name: sentry-postgres
    restart: always
    environment:
      POSTGRES_USER: "sentry"
      POSTGRES_PASSWORD: "sentry"
      POSTGRES_DB: "sentry"
#      TZ: "Asia/Tehran"
    volumes:
      - "pgdb_data:/var/lib/postgresql/data"
    depends_on:
      - "sentry-redis"
#--------------------------------------------------------------
# generate secret key
# docker run --rm sentry:9.1.2 config generate-secret-key

# docker run on-off command for initializeing databases and sentry
# docker run -it --rm -e SENTRY_SECRET_KEY='######' --link sentry-postgres:postgres --link sentry-redis:redis sentry:9.1.2 upgrade

  sentry-run:
    image: sentry:9.1.2
    container_name: sentry-run
    stdin_open: true
    tty: true
    links:
     - "sentry-redis"
     - "sentry-postgres"
    environment:
      SENTRY_SECRET_KEY: "#######"
      SENTRY_POSTGRES_HOST: "sentry-postgres"
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "sentry"
      SENTRY_REDIS_HOST: "sentry-redis"
      SENTRY_ADMIN_EMAIL: "root@localhost"
    depends_on:
      - "sentry-redis"
      - "sentry-postgres"
    command:
      - "upgrade"
#--------------------------------------------------------------
# user: root@localhost
# pass: ######
  sentry:
    image: sentry:9.1.2
    container_name: sentry
    restart: always
    links:
     - "sentry-redis"
     - "sentry-postgres"
    environment:
#      TZ: "Asia/Tehran"
      SENTRY_SECRET_KEY: "######"
      SENTRY_POSTGRES_HOST: "sentry-postgres"
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "sentry"
      SENTRY_REDIS_HOST: "sentry-redis"
      SENTRY_ADMIN_EMAIL: "root@localhost"
      SENTRY_ALLOW_REGISTRATION: "false"
      SENTRY_USE_REMOTE_USER: "true"
      SNUBA: "http://snuba-api:1218"
      SENTRY_SAMPLE_DATA: "false"
    depends_on:
      - "sentry-redis"
      - "sentry-postgres"
    volumes:
      - "sentry_data:/var/lib/sentry/files"
      - "sentry_config:/data"
      - "sentry_setting:/etc/sentry/"
#--------------------------------------------------------------
  cron:
    image: sentry:9.1.2
    container_name: cron
    restart: always
    links:
     - "sentry-redis"
     - "sentry-postgres"
    command: "sentry run cron"
    environment:
      SENTRY_SECRET_KEY: "######"
      SENTRY_POSTGRES_HOST: "sentry-postgres"
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "sentry"
      SENTRY_REDIS_HOST: "sentry-redis"
#      TZ: "Asia/Tehran"
    depends_on:
      - "sentry"
#--------------------------------------------------------------
  worker:
    image: sentry:9.1.2
    container_name: worker
    restart: always
    links:
     - "sentry-redis"
     - "sentry-postgres"
    command: "sentry run worker"
    environment:
      SENTRY_SECRET_KEY: "######"
      SENTRY_POSTGRES_HOST: "sentry-postgres"
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "sentry"
      SENTRY_REDIS_HOST: "sentry-redis"
#      TZ: "Asia/Tehran"
    depends_on:
      - "sentry"
#--------------------------------------------------------------
# FYI: refer to -->> https://docs.sentry.io/product/relay/getting-started/
#
# docker run --rm -it -v /mnt/storage/root-dir-docker/volumes/docker-services_relay_config/_data/:/work/.relay/ --entrypoint bash getsentry/relay:21.1.0 -c 'chown -R relay:relay /work/.relay'
# docker run --rm -it -v /mnt/storage/root-dir-docker/volumes/docker-services_relay_config/_data/:/work/.relay/ getsentry/relay:21.1.0 config init

  relay:
    image: getsentry/relay:21.1.0
    container_name: relay
    restart: always
    depends_on:
      - "sentry-redis"
#    environment:
#      TZ: "Asia/Tehran"
    volumes:
      - "relay_config:/work/.relay:rw"
      - "relay_geoip:/geoip"
#--------------------------------------------------------------
  clickhouse:
    image: yandex/clickhouse-server:21.1.2.15
    container_name: clickhouse
    restart: always
    environment:
#      TZ: "Asia/Tehran"
      MAX_MEMORY_USAGE_RATIO: 0.3
    volumes:
      - "clickhouse:/var/lib/clickhouse"
      - "clickhouse_log:/var/log/clickhouse-server"
      - "clickhouse_config:/etc/clickhouse-server/config.d"
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
#--------------------------------------------------------------
  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.0
    container_name: zookeeper
    restart: always
    environment:
#      TZ: "Asia/Tehran"
      ZOOKEEPER_CLIENT_PORT: "2181"
      CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
      ZOOKEEPER_LOG4J_ROOT_LOGLEVEL: "WARN"
      ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL: "WARN"
    volumes:
        - "zookeeper:/var/lib/zookeeper/data"
        - "zookeeper_log:/var/lib/zookeeper/log"
        - "zookeeper_secrets:/etc/zookeeper/secrets"
#--------------------------------------------------------------
  kafka:
    image: confluentinc/cp-kafka:5.5.0
    container_name: kafka
    restart: always
    depends_on:
      - "zookeeper"
    environment:
#      TZ: "Asia/Tehran"
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
      KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
      KAFKA_LOG_RETENTION_HOURS: "24"
      KAFKA_MESSAGE_MAX_BYTES: "50000000" #50MB or bust
      KAFKA_MAX_REQUEST_SIZE: "50000000" #50MB on requests apparently too
      CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
      KAFKA_LOG4J_LOGGERS: "kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN"
      KAFKA_LOG4J_ROOT_LOGLEVEL: "WARN"
      KAFKA_TOOLS_LOG4J_LOGLEVEL: "WARN"
    volumes:
      - "kafka:/var/lib/kafka/data"
      - "kafka_log:/var/lib/kafka/log"
      - "kafka_settings:/etc/kafka"
#--------------------------------------------------------------
  snuba-api:
    image: getsentry/snuba:21.1.0
    container_name: snuba-api
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
#      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
#      UWSGI_MAX_REQUESTS: "10000"
#      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
#--------------------------------------------------------------
#docker-compose exec -- kafka kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic events
  snuba-consumer:
    image: getsentry/snuba:21.1.0
    container_name: snuba-consumer
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
#--------------------------------------------------------------
  snuba-outcomes-consumer:
    image: getsentry/snuba:21.1.0
    container_name: snuba-outcomes-consumer
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
#--------------------------------------------------------------
  snuba-sessions-consumer:
    image: getsentry/snuba:21.1.0
    container_name: snuba-sessions-consumer
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: consumer --storage sessions_raw --auto-offset-reset=latest --max-batch-time-ms 750
#--------------------------------------------------------------
  snuba-transactions-consumer:
    image: getsentry/snuba:21.1.0
    container_name: snuba-transactions-consumer
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: consumer --storage transactions --consumer-group transactions_group --auto-offset-reset=latest --max-batch-time-ms 750 --commit-log-topic=snuba-commit-log
#--------------------------------------------------------------
  snuba-replacer:
    image: getsentry/snuba:21.1.0
    container_name: snuba-replacer
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: replacer --storage events --auto-offset-reset=latest --max-batch-size 3
#--------------------------------------------------------------
  snuba-subscription-consumer-events:
    image: getsentry/snuba:21.1.0
    container_name: snuba-subscription-consumer-events
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: subscriptions --auto-offset-reset=latest --consumer-group=snuba-events-subscriptions-consumers --topic=events --result-topic=events-subscription-results --dataset=events --commit-log-topic=snuba-commit-log --commit-log-group=snuba-consumers --delay-seconds=60 --schedule-ttl=60
#--------------------------------------------------------------
  snuba-subscription-consumer-transactions:
    image: getsentry/snuba:21.1.0
    container_name: snuba-subscription-consumer-transactions
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: subscriptions --auto-offset-reset=latest --consumer-group=snuba-transactions-subscriptions-consumers --topic=events --result-topic=transactions-subscription-results --dataset=transactions --commit-log-topic=snuba-commit-log --commit-log-group=transactions_group --delay-seconds=60 --schedule-ttl=60
#--------------------------------------------------------------
  snuba-cleanup:
    image: getsentry/snuba:21.1.0
    container_name: snuba-cleanup
    restart: always
    depends_on:
      - "sentry-redis"
      - "clickhouse"
      - "kafka"
    environment:
#      TZ: "Asia/Tehran"
      SNUBA_SETTINGS: docker
      CLICKHOUSE_HOST: clickhouse
      DEFAULT_BROKERS: "kafka:9092"
      REDIS_HOST: "sentry-redis"
      UWSGI_MAX_REQUESTS: "10000"
      UWSGI_DISABLE_LOGGING: "true"
      SENTRY_EVENT_RETENTION_DAYS: 30
    command: '"*/5 * * * * gosu snuba snuba cleanup --dry-run False"'
#--------------------------------------------------------------
  symbolicator:
    image: getsentry/symbolicator:nightly
    container_name: symbolicator
    restart: always
    volumes:
      - "sentry_symbolicator:/etc/symbolicator"
    command: run -c /etc/symbolicator/config.yml
#--------------------------------------------------------------
  symbolicator-cleanup:
    image: getsentry/snuba:21.1.0
    container_name: symbolicator-cleanup
    restart: always
    command: '"55 23 * * * gosu symbolicator symbolicator cleanup"'
    volumes:
      - "sentry_symbolicator:/data"
#--------------------------------------------------------------

mj.taleghani · July 4, 2021, 4:55am

there was my service running. but not properly. my question is how did you can help other requesters that provided you with more info with their problem?!
What I saw was only a line by you that wrote any update or just saying update the repo and run docker again!!!!
these are not supports.

BYK · July 6, 2021, 9:51am

Mind you, Sentry self-hosted is offered as is without any support or guarantees. If you don’t want to deal with these cases or need dedicated support, you can always stop using self-hosted and/or start using our cloud hosted service. You can read more about self-hosted support here: https://develop.sentry.dev/self-hosted/support/

Now, with that out of the way, looks like you are changing your Relay’s mode. It should run in managed mode, which is the default and I’m quite sure that’s the reason for your issues. Can you try removing that mode: proxy from your relay config and restarting it?

system · October 4, 2021, 9:52am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Sentry-relay "project state for key X is missing key id" when sentry-relay on separate subdomain On-Premise	5	2523	May 19, 2021
Relay errors with exist key id On-Premise	2	1600	April 5, 2021
Installation problems running the default install.sh script On-Premise	7	2312	August 5, 2021
Could not write /etc/relay/credentials.json On-Premise	3	2953	April 27, 2020
Can't access Sentry UI after installation On-Premise	12	4284	December 8, 2020

Relay missing key id project

Related topics