Decompressing gzipped json blob-as-text (nodestore_node table) data from on-premise instance of Sentry

erica · November 23, 2020, 10:40pm

we have an on-premise Sentry instance and we need to export all the event data. i’ve been able to export the tables that we are interested in and most are readable with the exception of the nodestore_node table. i think the data are stored as gzipped json blob-as-text, which i’m not familiar with. i’ve tried a bunch of different things but haven’t been successful in converting to something human readable. a few things i’ve tried so far:

SQL uncompress (just ended up with null for each row of converted_data)
SELECT nodestore_node.data, CONVERT( UNCOMPRESS( nodestore_node.data ) USING 'utf8' ) AS converted_data FROM nodestore_node;
SQL cast (got error messages about CAST for this one)
SELECT CAST( CAST( 'string' as XML ).value('.','varbinary(max)') AS varchar(max) )
decompress gzip in JS

is there a way that i could convert the blobs to something human readable? or is there an API to pull all the converted nodestore data?

any ideas would be greatly appreciated!! thank you!

BYK · November 24, 2020, 11:02am

Is this for back-up/restore or migration purposes? If so, instead of extracting event data, you should just be able to use the database files directly. Otherwise I think this api endpoint would help you more: https://docs.sentry.io/api/events/list-a-projects-events/

erica · November 24, 2020, 6:00pm

that would work great. i tried using the DSN authentication but that didn’t work. what’s the auth endpoint to generate a token locally?

thank you!

BYK · November 24, 2020, 6:54pm

Would this help: Authentication | Sentry Documentation

We also do something similar here in our onpremise tests, that said if you find that ugly and hard to decipher, I wouldn’t blame you

github.com

getsentry/self-hosted/blob/19f4561a9e2abe32dc5eb5a03a332b50f2265b4b/test.sh#L77-L107


      
          SENTRY_DSN=$(sentry_api_request "projects/sentry/internal/keys/" | awk 'BEGIN { RS=",|:{\n"; FS="\""; } $2 == "public" && $4 ~ "^http" { print $4; exit; }')
          # We ignore the protocol and the host as we already know those
          DSN_PIECES=(`echo $SENTRY_DSN | sed -ne 's|^https\?://\([0-9a-z]\+\)@[^/]\+/\([0-9]\+\)$|\1\n\2|p'`)
          SENTRY_KEY=${DSN_PIECES[0]}
          PROJECT_ID=${DSN_PIECES[1]}
          
          TEST_EVENT_ID=$(export LC_ALL=C; head /dev/urandom | tr -dc "a-f0-9" | head -c 32)
          # Thanks @untitaker - https://forum.sentry.io/t/how-can-i-post-with-curl-a-sentry-event-which-authentication-credentials/4759/2?u=byk
          echo "Creating test event..."
          curl -sf --data '{"event_id": "'"$TEST_EVENT_ID"'","level":"error","message":"a failure","extra":{"object":"42"}}' -H 'Content-Type: application/json' -H "X-Sentry-Auth: Sentry sentry_version=7, sentry_key=$SENTRY_KEY, sentry_client=test-bash/0.1" "$SENTRY_TEST_HOST/api/$PROJECT_ID/store/" -o /dev/null
          
          EVENT_PATH="projects/sentry/internal/events/$TEST_EVENT_ID/"
          export -f sentry_api_request get_csrf_token
          export SENTRY_TEST_HOST COOKIE_FILE EVENT_PATH
          printf "Getting the test event back"
          timeout 30 bash -c 'until $(sentry_api_request "$EVENT_PATH" -Isf -X GET -o /dev/null); do printf '.'; sleep 0.5; done'
          echo "";
          
          EVENT_RESPONSE=$(sentry_api_request "$EVENT_PATH")
          declare -a EVENT_TEST_STRINGS=(

This file has been truncated. show original

erica · November 25, 2020, 11:34pm

thank you!!

i think the auth token documentation (https://docs.sentry.io/api/auth/#auth-tokens) is for projects hosted by sentry io and not on-premise. but let me know if i’m missing something!

thanks for sharing the test - i ran the script line by line in my terminal, with updates for our environment set-up, but it just hangs on:

github.com

getsentry/onpremise/blob/19f4561a9e2abe32dc5eb5a03a332b50f2265b4b/test.sh#L92


TEST_EVENT_ID=$(export LC_ALL=C; head /dev/urandom | tr -dc "a-f0-9" | head -c 32)
# Thanks @untitaker - https://forum.sentry.io/t/how-can-i-post-with-curl-a-sentry-event-which-authentication-credentials/4759/2?u=byk
echo "Creating test event..."
curl -sf --data '{"event_id": "'"$TEST_EVENT_ID"'","level":"error","message":"a failure","extra":{"object":"42"}}' -H 'Content-Type: application/json' -H "X-Sentry-Auth: Sentry sentry_version=7, sentry_key=$SENTRY_KEY, sentry_client=test-bash/0.1" "$SENTRY_TEST_HOST/api/$PROJECT_ID/store/" -o /dev/null
EVENT_PATH="projects/sentry/internal/events/$TEST_EVENT_ID/"
export -f sentry_api_request get_csrf_token
export SENTRY_TEST_HOST COOKIE_FILE EVENT_PATH
printf "Getting the test event back"
timeout 30 bash -c 'until $(sentry_api_request "$EVENT_PATH" -Isf -X GET -o /dev/null); do printf '.'; sleep 0.5; done'
echo "";
EVENT_RESPONSE=$(sentry_api_request "$EVENT_PATH")
declare -a EVENT_TEST_STRINGS=(
  '"eventID":"'"$TEST_EVENT_ID"'"'
  '"message":"a failure"'
  '"title":"a failure"'
  '"object":"42"'
)
for i in "${EVENT_TEST_STRINGS[@]}"

i also tried running the top part of the test script (again with environment updates) but got a CSRF error for the login.

any easy way you can think to generate a token locally?

thanks, again!

BYK · November 26, 2020, 7:09pm

It applies to both as they use the same codebase. Just replace sentry.io parts with your own instance’s URL.

You don’t need the event generation parts. This was just to give you an idea about how to interact with the API, that said it is using cookie authentication so probably not very appropriate for your use case.

erica · November 30, 2020, 10:19pm

ohhh - amazing!!! thank you so much !

Topic		Replies	Views
Postgres nodestore_node table 124gb On-Premise	3	7698	May 27, 2021
Postgres is growing rapidly - Manual cleanup On-Premise	17	13363	September 16, 2021
Base64 encoded Source Maps support	4	3309	December 22, 2016
Nodestore_node size On-Premise	6	3093	March 23, 2021
Possible to send more complex (unstructured) data to Sentry?	8	2443	May 13, 2021

Decompressing gzipped json blob-as-text (nodestore_node table) data from on-premise instance of Sentry

Related topics