Decompressing gzipped json blob-as-text (nodestore_node table) data from on-premise instance of Sentry

erica · November 23, 2020, 10:41pm

we have an on-premise Sentry instance and we need to export all the event data. i’ve been able to export the tables that we are interested in and most are readable with the exception of the nodestore_node table. i think the data are stored as gzipped json blob-as-text, which i’m not familiar with. i’ve tried a bunch of different things but haven’t been successful in converting to something human readable. a few things i’ve tried so far:

SQL uncompress (just ended up with null for each row of converted_data)
SELECT nodestore_node.data, CONVERT( UNCOMPRESS( nodestore_node.data ) USING 'utf8' ) AS converted_data FROM nodestore_node;
SQL cast (got error messages about CAST for this one)
SELECT CAST( CAST( 'string' as XML ).value('.','varbinary(max)') AS varchar(max) )
decompress gzip in JS

is there a way that i could convert the blobs to something human readable? or is there an API to pull all the converted nodestore data?

any ideas would be greatly appreciated!! thank you!

BYK · November 24, 2020, 11:02am

Is this for back-up/restore or migration purposes? If so, instead of extracting event data, you should just be able to use the database files directly. Otherwise I think this api endpoint would help you more: https://docs.sentry.io/api/events/list-a-projects-events/

erica · November 24, 2020, 6:00pm

that would work great. i tried using the DSN authentication but that didn’t work. what’s the auth endpoint to generate a token locally?

thank you!

BYK · November 30, 2020, 10:17pm

Would this help: https://docs.sentry.io/api/auth/#auth-tokens

We also do something similar here in our onpremise tests, that said if you find that ugly and hard to decipher, I wouldn’t blame you

github.com

getsentry/onpremise/blob/19f4561a9e2abe32dc5eb5a03a332b50f2265b4b/test.sh#L77-L107


SENTRY_DSN=$(sentry_api_request "projects/sentry/internal/keys/" | awk 'BEGIN { RS=",|:{\n"; FS="\""; } $2 == "public" && $4 ~ "^http" { print $4; exit; }')
# We ignore the protocol and the host as we already know those
DSN_PIECES=(`echo $SENTRY_DSN | sed -ne 's|^https\?://\([0-9a-z]\+\)@[^/]\+/\([0-9]\+\)$|\1\n\2|p'`)
SENTRY_KEY=${DSN_PIECES[0]}
PROJECT_ID=${DSN_PIECES[1]}
TEST_EVENT_ID=$(export LC_ALL=C; head /dev/urandom | tr -dc "a-f0-9" | head -c 32)
# Thanks @untitaker - https://forum.sentry.io/t/how-can-i-post-with-curl-a-sentry-event-which-authentication-credentials/4759/2?u=byk
echo "Creating test event..."
curl -sf --data '{"event_id": "'"$TEST_EVENT_ID"'","level":"error","message":"a failure","extra":{"object":"42"}}' -H 'Content-Type: application/json' -H "X-Sentry-Auth: Sentry sentry_version=7, sentry_key=$SENTRY_KEY, sentry_client=test-bash/0.1" "$SENTRY_TEST_HOST/api/$PROJECT_ID/store/" -o /dev/null
EVENT_PATH="projects/sentry/internal/events/$TEST_EVENT_ID/"
export -f sentry_api_request get_csrf_token
export SENTRY_TEST_HOST COOKIE_FILE EVENT_PATH
printf "Getting the test event back"
timeout 30 bash -c 'until $(sentry_api_request "$EVENT_PATH" -Isf -X GET -o /dev/null); do printf '.'; sleep 0.5; done'
echo "";
EVENT_RESPONSE=$(sentry_api_request "$EVENT_PATH")
declare -a EVENT_TEST_STRINGS=(

This file has been truncated. show original

erica · November 25, 2020, 11:36pm

thank you!!

i think the auth token documentation (https://docs.sentry.io/api/auth/#auth-tokens) is for projects hosted by sentry io and not on-premise. but let me know if i’m missing something!

thanks for sharing the test - i ran the script line by line in my terminal, with updates for our environment set-up, but it just hangs on:

github.com

getsentry/onpremise/blob/19f4561a9e2abe32dc5eb5a03a332b50f2265b4b/test.sh#L92


TEST_EVENT_ID=$(export LC_ALL=C; head /dev/urandom | tr -dc "a-f0-9" | head -c 32)
# Thanks @untitaker - https://forum.sentry.io/t/how-can-i-post-with-curl-a-sentry-event-which-authentication-credentials/4759/2?u=byk
echo "Creating test event..."
curl -sf --data '{"event_id": "'"$TEST_EVENT_ID"'","level":"error","message":"a failure","extra":{"object":"42"}}' -H 'Content-Type: application/json' -H "X-Sentry-Auth: Sentry sentry_version=7, sentry_key=$SENTRY_KEY, sentry_client=test-bash/0.1" "$SENTRY_TEST_HOST/api/$PROJECT_ID/store/" -o /dev/null
EVENT_PATH="projects/sentry/internal/events/$TEST_EVENT_ID/"
export -f sentry_api_request get_csrf_token
export SENTRY_TEST_HOST COOKIE_FILE EVENT_PATH
printf "Getting the test event back"
timeout 30 bash -c 'until $(sentry_api_request "$EVENT_PATH" -Isf -X GET -o /dev/null); do printf '.'; sleep 0.5; done'
echo "";
EVENT_RESPONSE=$(sentry_api_request "$EVENT_PATH")
declare -a EVENT_TEST_STRINGS=(
  '"eventID":"'"$TEST_EVENT_ID"'"'
  '"message":"a failure"'
  '"title":"a failure"'
  '"object":"42"'
)
for i in "${EVENT_TEST_STRINGS[@]}"

i also tried running the top part of the test script (again with environment updates) but got a CSRF error for the login.

any easy way you can think to generate a token locally?

thanks, again!

BYK · November 26, 2020, 7:09pm

It applies to both as they use the same codebase. Just replace sentry.io parts with your own instance’s URL.

You don’t need the event generation parts. This was just to give you an idea about how to interact with the API, that said it is using cookie authentication so probably not very appropriate for your use case.

erica · November 30, 2020, 10:19pm

ohhh - amazing!!! thank you so much !