After upgrade from 9.1.2 to 10.0/20.9/latest no events details

mitmafiy · October 13, 2020, 7:17pm

Hi,
I have sentry 9.1.2, deploy in docker from onpromice and i will need uprgade sentry to 20.9.
After upgrade from 9.1.2 to 10.0 or 20.9/latest, I receive the same one error when to open the details of an events:

Now I upgrade sentry 9.1.2 to 10.0.0. After upgrade, all projects, issues and users all right.

But when I try open events details, I have problem described above and see error from Sentry web instances themselves:

migration log clickhouse:

exec gosu snuba snuba migrate
2020-10-13 14:02:05,957 Migrating dataset outcomes_raw
2020-10-13 14:02:05,972 Executing migration: ALTER TABLE outcomes_raw_local ADD COLUMN size Nullable(UInt32)
2020-10-13 14:02:05,977 Column ‘size’ exists in local ClickHouse but not in schema!
2020-10-13 14:02:05,978 Migrating dataset events
2020-10-13 14:02:05,986 Migrating dataset groupassignee
2020-10-13 14:02:05,992 Migrating dataset transactions
2020-10-13 14:02:05,996 Column ‘transaction_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
2020-10-13 14:02:05,996 Column ‘transaction_status’ type differs between local ClickHouse and schema! (expected: UInt8 DEFAULT 2, is: UInt8)
2020-10-13 14:02:05,996 Column ‘_start_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(start_ts), is: Date)
2020-10-13 14:02:05,996 Column ‘_finish_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(finish_ts), is: Date)
2020-10-13 14:02:05,996 Column ‘user’ type differs between local ClickHouse and schema! (expected: String DEFAULT ‘’, is: String)
2020-10-13 14:02:05,996 Column ‘user_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
2020-10-13 14:02:05,996 Column ‘sdk_name’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
2020-10-13 14:02:05,996 Column ‘sdk_version’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
2020-10-13 14:02:05,999 Column ‘transaction_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
2020-10-13 14:02:05,999 Column ‘transaction_status’ type differs between local ClickHouse and schema! (expected: UInt8 DEFAULT 2, is: UInt8)
2020-10-13 14:02:05,999 Column ‘_start_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(start_ts), is: Date)
2020-10-13 14:02:05,999 Column ‘_finish_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(finish_ts), is: Date)
2020-10-13 14:02:05,999 Column ‘user’ type differs between local ClickHouse and schema! (expected: String DEFAULT ‘’, is: String)
2020-10-13 14:02:05,999 Column ‘user_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
2020-10-13 14:02:05,999 Column ‘sdk_name’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
2020-10-13 14:02:05,999 Column ‘sdk_version’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
2020-10-13 14:02:05,999 Migrating dataset groupedmessage
2020-10-13 14:02:06,005 Migrating dataset discover2020-10-13 14:02:06,006 Migrating dataset outcomes
2020-10-13 14:02:06,009 Column ‘size’ exists in local ClickHouse but not in schema!

migration log postgresql:

14:02:14 [INFO] sentry.plugins.github: apps-not-configured
Faking migration for contenttypes.0001_initial
Faking migration for contenttypes.0002_remove_content_type_name
Faking migration for auth.0001_initial
Faking migration for auth.0002_alter_permission_name_max_length
Faking migration for auth.0003_alter_user_email_max_length
Faking migration for auth.0004_alter_user_username_opts
Faking migration for auth.0005_alter_user_last_login_null
Faking migration for auth.0006_require_contenttypes_0002
Faking migration for sites.0001_initial
Faking migration for admin.0001_initial
Faking migration for sessions.0001_initial
^[[36;1mOperations to perform:^[[0m
^[[1m Apply all migrations: ^[[0mnodestore, sessions, admin, sentry, jira_ac, sites, auth, social_auth, contenttypes
^[[36;1mRunning migrations:^[[0m
Rendering model states…^[[32;1m DONE^[[0m
Applying admin.0002_logentry_remove_auto_add…^[[32;1m OK^[[0m
Applying auth.0007_alter_validators_add_error_messages…^[[32;1m OK^[[0m
Applying sentry.0002_912_to_recent…^[[32;1m OK^[[0m
Applying sentry.0003_auto_20191022_0122…^[[32;1m OK^[[0m
Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau…^[[32;1m OK^[[0m
Applying sentry.0005_fix_content_types…^[[32;1m OK^[[0m
Applying sentry.0006_sentryapp_date_published…^[[32;1m OK^[[0m
Applying sentry.0007_auto_20191029_0131…^[[32;1m OK^[[0m
Applying sentry.0008_auto_20191030_0016…^[[32;1m OK^[[0m
Applying sentry.0009_auto_20191101_1608…^[[32;1m OK^[[0m
Applying sentry.0010_auto_20191104_1641…^[[32;1m OK^[[0m
Applying sentry.0011_remove_pagerdutyservice_service_id_from_state…^[[32;1m OK^[[0m
Applying sentry.0012_remove_pagerdutyservice_service_id…^[[32;1m OK^[[0m
Applying sentry.0013_auto_20191111_1829…^[[32;1m OK^[[0m
Applying sentry.0014_delete_sentryappwebhookerror…^[[32;1m OK^[[0m
Applying sentry.0015_delete_sentryappwebhookerror_db…^[[32;1m OK^[[0m
Applying sentry.0016_delete_alert_rule_deprecated_fields…^[[32;1m OK^[[0m
Applying sentry.0017_incident_aggregation…^[[32;1m OK^[[0m
Applying sentry.0018_discoversavedquery_version…^[[32;1m OK^[[0m
Applying sentry.0019_auto_20191114_2040…^[[32;1m OK^[[0m
Applying sentry.0020_auto_20191125_1420…^[[32;1m OK^[[0m
Applying sentry.0021_auto_20191203_1803…^[[32;1m OK^[[0m
Applying sentry.0021_auto_20191202_1716…^[[32;1m OK^[[0m Applying sentry.0022_merge…^[[32;1m OK^[[0m
Applying sentry.0023_hide_environment_none_20191126…^[[32;1m OK^[[0m
Applying sentry.0024_auto_20191230_2052…Events to process: 66071
Done

any help would be helpful

BYK · October 14, 2020, 9:20am

This part of the logs suggest that your migrations got interrupted in the middle. Possibly due to an OOM kill. I’d try rerunning the migrations with more RAM available.

Also I strongly recommend trying to upgrade to a more recent version rather than 10.0.0 (like 20.9.0 or the upcoming 20.10.0 which will be released on Thursday) as they have important bugfixes to make this migration easier.

mitmafiy · October 14, 2020, 10:17am

Thanks for your reply.
VM RAM - 12GB.
I tried directly upgrade from 9.1.2 to 20.9 or latest and I have the same problem with the events details display. Then I started trying to upgrade versions sequentially and couldn’t upgrade correctly from 9.1.2 to 10.0.
Okay, I’ll wait 20.10.0 and I will try upgrade to this version.

BYK · October 14, 2020, 8:48pm

This may still be exhausted. Are you able to check OOM killer logs?

mitmafiy · October 15, 2020, 7:02am

Yes, I checked logs, but are no OOMk events in the logs.

mitmafiy · October 15, 2020, 7:22am

Now I have postgresql v9.6. Before to upgrade sentry to 20.10, I need upgrade postgresql to 12 or 13?

BYK · October 15, 2020, 8:18am

We still use Postgres 9.6:

github.com

getsentry/onpremise/blob/1399be6a68f3adda8f039c4ba463dea1db9e6932/docker-compose.yml#L64-L66


postgres:
  << : *restart_policy
  image: 'postgres:9.6'

mitmafiy · October 16, 2020, 9:11am

Today, I uprgade 9.1.2 to 20.10, but unfortunately the problem remained.

migration log postgresql:

08:11:55 [INFO] sentry.plugins.github: apps-not-configured
Faking migration for contenttypes.0001_initial
Faking migration for contenttypes.0002_remove_content_type_name
Faking migration for auth.0001_initial
Faking migration for auth.0002_alter_permission_name_max_length
Faking migration for auth.0003_alter_user_email_max_length
Faking migration for auth.0004_alter_user_username_opts
Faking migration for auth.0005_alter_user_last_login_null
Faking migration for auth.0006_require_contenttypes_0002
Faking migration for sites.0001_initial
Faking migration for admin.0001_initial
Faking migration for sessions.0001_initial
Operations to perform:
Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying admin.0002_logentry_remove_auto_add… OK
Applying auth.0007_alter_validators_add_error_messages… OK
Applying auth.0008_alter_user_username_max_length… OK
Applying sentry.0002_912_to_recent… OK
Applying sentry.0003_auto_20191022_0122… OK
Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau… OK
Applying sentry.0005_fix_content_types… OK
Applying sentry.0006_sentryapp_date_published… OK
Applying sentry.0007_auto_20191029_0131… OK
Applying sentry.0008_auto_20191030_0016… OK
Applying sentry.0009_auto_20191101_1608… OK
Applying sentry.0010_auto_20191104_1641… OK
Applying sentry.0011_remove_pagerdutyservice_service_id_from_state… OK
Applying sentry.0012_remove_pagerdutyservice_service_id… OK
Applying sentry.0013_auto_20191111_1829… OK
Applying sentry.0014_delete_sentryappwebhookerror… OK
Applying sentry.0015_delete_sentryappwebhookerror_db… OK
Applying sentry.0016_delete_alert_rule_deprecated_fields… OK
Applying sentry.0017_incident_aggregation… OK
Applying sentry.0018_discoversavedquery_version… OK
Applying sentry.0019_auto_20191114_2040… OK
Applying sentry.0020_auto_20191125_1420… OK
Applying sentry.0021_auto_20191203_1803… OK
Applying sentry.0021_auto_20191202_1716… OK
Applying sentry.0022_merge… OK
Applying sentry.0023_hide_environment_none_20191126… OK
Applying sentry.0024_auto_20191230_2052…Events to process: 62444

Event migration done. Migrated 62444 of 62444 events.

OK
Applying sentry.0025_organizationaccessrequest_requester… OK
Applying sentry.0026_delete_event… OK
Applying sentry.0027_exporteddata… OK
Applying sentry.0028_user_reports… OK
Applying sentry.0029_discover_query_upgrade… OK
Applying sentry.0030_auto_20200201_0039… OK
Applying sentry.0031_delete_alert_rules_and_incidents… OK
User Options: 100% |#############################################| Time: 0:00:00
OK
Applying sentry.0033_auto_20200210_2137… OK
Applying sentry.0034_auto_20200210_2311… OK
Applying sentry.0035_auto_20200127_1711… OK
Applying sentry.0036_auto_20200213_0106… OK
Applying sentry.0037_auto_20200213_0140… OK
Applying sentry.0038_auto_20200213_1904… OK
Applying sentry.0039_delete_incidentsuspectcommit… OK
Applying sentry.0040_remove_incidentsuspectcommittable… OK
Applying sentry.0041_incidenttrigger_date_modified… OK
Applying sentry.0042_auto_20200214_1607… OK
Applying sentry.0043_auto_20200218_1903… OK
Applying sentry.0044_auto_20200219_0018… OK
Applying sentry.0045_remove_incidentactivity_event_stats_snapshot… OK
Applying sentry.0046_auto_20200221_1735… OK
Applying sentry.0047_auto_20200224_2319… OK
Applying sentry.0048_auto_20200302_1825… OK
Applying sentry.0049_auto_20200304_0254… OK
Applying sentry.0050_auto_20200306_2346… OK
Audit Log Entrys: N/A% | | ETA: --:–:--None
08:25:59 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
None
…
08:26:00 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
Audit Log Entrys: 100% |#########################################| Time: 0:00:01
OK
Applying sentry.0052_organizationonboardingtask_completion_seen… OK
Applying sentry.0053_migrate_alert_task_onboarding… OK
Applying sentry.0054_create_key_transaction… OK
Applying sentry.0055_query_subscription_status… OK
Applying sentry.0056_remove_old_functions… OK
Applying sentry.0057_remove_unused_project_flag… OK
Applying sentry.0058_project_issue_alerts_targeting… OK
Applying sentry.0059_add_new_sentry_app_features… OK
Applying sentry.0060_add_file_eventattachment_index… OK
Applying sentry.0061_alertrule_partial_index… OK
Applying sentry.0062_key_transactions_unique_with_owner… OK
Applying sentry.0063_drop_alertrule_constraint… OK
Applying sentry.0064_project_has_transactions… OK
Applying sentry.0065_add_incident_status_method… OK
Applying sentry.0066_alertrule_manager… OK
Organizations: 100% |############################################| Time: 0:00:00
OK
Applying sentry.0068_project_default_flags… OK
Applying sentry.0069_remove_tracked_superusers… OK
Applying sentry.0070_incident_snapshot_support… OK
Applying sentry.0071_add_default_fields_model_subclass… OK
Applying sentry.0072_alert_rules_query_changes… OK
Applying sentry.0073_migrate_alert_query_model… OK
Applying sentry.0074_add_metric_alert_feature… OK
Applying sentry.0075_metric_alerts_fix_releases… OK
Applying sentry.0076_alert_rules_disable_constraints… OK
Applying sentry.0077_alert_query_col_drop_state… OK
Applying sentry.0078_incident_field_updates… OK
Applying sentry.0079_incidents_remove_query_field_state… OK
Applying sentry.0080_alert_rules_drop_unused_tables_cols… OK
Applying sentry.0081_add_integraiton_upgrade_audit_log… OK
Applying sentry.0082_alert_rules_threshold_float… OK
Applying sentry.0083_add_max_length_webhook_url… OK
Applying sentry.0084_exported_data_blobs… OK
Applying sentry.0085_fix_error_rate_snuba_query… OK
Applying sentry.0086_sentry_app_installation_for_provider… OK
Applying sentry.0087_fix_time_series_data_type… OK
Applying sentry.0088_rule_level_resolve_threshold_type… OK
Applying sentry.0089_rule_level_fields_backfill… OK
Audit Log Entrys: N/A% | | ETA: --:–:--None
08:26:16 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
None
…
08:26:17 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
Audit Log Entrys: 100% |#########################################| Time: 0:00:01
OK
Applying sentry.0091_alertruleactivity… OK
Applying sentry.0092_remove_trigger_threshold_type_nullable… OK
Applying sentry.0093_make_identity_user_id_textfield… OK
Applying sentry.0094_cleanup_unreferenced_event_files… OK
Applying sentry.0095_ruleactivity… OK
Applying sentry.0096_sentry_app_component_skip_load_on_open… OK
Applying sentry.0097_add_sentry_app_id_to_sentry_alertruletriggeraction… OK
Applying sentry.0098_add-performance-onboarding… OK
Projects: 100% |#################################################| Time: 0:00:00
OK
Applying sentry.0100_file_type_on_event_attachment… OK
Applying sentry.0101_backfill_file_type_on_event_attachment… OK
Applying sentry.0102_collect_relay_analytics… OK
Applying sentry.0103_project_has_alert_filters… OK
Applying sentry.0104_collect_relay_public_key_usage… OK
Applying sentry.0105_remove_nullability_of_event_attachment_type… OK
Applying sentry.0106_service_hook_project_id_nullable… OK
Applying sentry.0107_remove_spaces_from_slugs… OK
Applying sentry.0108_update_fileblob_action… OK
Applying sentry.0109_sentry_app_creator… OK
Applying sentry.0110_sentry_app_creator_backill… OK
Applying sentry.0111_snuba_query_event_type… OK
Applying sentry.0112_groupinboxmodel… OK
Applying sentry.0113_add_repositoryprojectpathconfig… OK
Applying sites.0002_alter_domain_unique… OK

error when i opens events details:

error from Sentry web instances themselves:

docker-compouse log:

nginx_1 | 192.168.100.65 - - [16/Oct/2020:10:27:40 +0000] “GET /api/0/issues/2464/ HTTP/1.1” 200 3317 “http://192.168.100.74:9000/organizations/sentry/issues/2464/?project=17&query=is%3Aunresolved” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0” “-”
web_1 | 10:27:40 [WARNING] django.request: Not Found: /api/0/issues/2464/events/latest/ (status_code=404 request=<WSGIRequest: GET u’/api/0/issues/2464/events/latest/’>)
nginx_1 | 192.168.100.65 - - [16/Oct/2020:10:27:40 +0000] “GET /api/0/issues/2464/events/latest/ HTTP/1.1” 404 28 “http://192.168.100.74:9000/organizations/sentry/issues/2464/?project=17&query=is%3Aunresolved” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0” “-”
nginx_1 | 2020/10/16 10:27:40 [warn] 6#6: *274 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000015, client: 172.21.0.1, server: , request: “POST /api/1/store/?sentry_key=a4fd59524c223ce27b701&sentry_version=7 HTTP/1.0”, host: “http://192.168.100.74:9000/”
nginx_1 | 172.21.0.1 - - [16/Oct/2020:10:27:40 +0000] “POST /api/1/store/?sentry_key=a4fd59524c223ce27b701&sentry_version=7 HTTP/1.0” 200 41 “http://192.168.100.74:9000/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0” “192.168.100.65”
ingest-consumer_1 | 10:27:41 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u’ingest-events’, 0): [18L, 18L]}): forced:False size:False time:True
ingest-consumer_1 | 10:27:41 [INFO] batching-kafka-consumer: Worker flush took 5ms
nginx_1 | 172.21.0.1 - - [16/Oct/2020:10:27:42 +0000] “GET /_static/1602843379/sentry/dist/vendor.js.map HTTP/1.1” 200 1661333 “-” “sentry/20.10.0 (https://sentry.io)” “-”
snuba-transactions-consumer_1 | 2020-10-16 10:27:43,532 Completed processing <Batch: 1 message, open for 1.00 seconds>.
snuba-outcomes-consumer_1 | 2020-10-16 10:27:43,539 Completed processing <Batch: 1 message, open for 1.01 seconds>.
snuba-consumer_1 | 2020-10-16 10:27:43,549 Completed processing <Batch: 1 message, open for 1.02 seconds>.

mitmafiy · October 18, 2020, 5:36pm

I looked in table “nodestore_node”:

          id                
----------------------------------±------------------------------
20b661d14d9ea95ebb3c606ca84a1281
2c1b220825cda454a73937368ce71fb8
1f29041a7ff0932ec1c85c68313f2fb0
87b0b320a2238c0049b92a09dab0ad52
ac64e478e332eabe9ddb3908d929c2e9
AyVG1I5aQcSoBK0V8iLukQ==
He/BHxcfSHixscnM9xGJ8g==
qPIBpNj0QpSa9oCwPY9ixg==
eKXnBhfQTVeHPVkNhtcZmA==

format id before upgrade: AyVG1I5aQcSoBK0V8iLukQ==
format id after upgrade: 20b661d14d9ea95ebb3c606ca84a1281

perhaps that’s the problem?

mitmafiy · October 19, 2020, 11:16am

apparently, this ID changed when I updated my sentry 8.15 to 9.1.2. and on v9.1.2, I can see that my events/issues are shown correctly.
when I updated 9.1.2 to 20.x, I see that query to detailes is executed using id with the UUID format. But old events detailes has old format id.

how do you think this can be fixed?

BYK · October 19, 2020, 4:56pm

How old are these events?

mitmafiy · October 19, 2020, 5:45pm

they’re not old. I just week ago upgraded sentry to 9.1.2. Our team long worked on v8.15)

BYK · October 19, 2020, 6:18pm

Ah, sorry I misunderstood. Okay investigating this still.

BYK · October 19, 2020, 6:40pm

@mitmafiy - thanks a ton for the detailed information you have shared with us! I think you have helped us identify a long-standing migration issue. We are currently actively investigating this and will be reporting back here.

BYK · October 22, 2020, 9:02am

@mitmafiy we have a potential fix in place but the fix needs to run with your old data. If you can do it, please run the following in the 20.10.1 checkout of the onpremise repo:

SENTRY_IMAGE=us.gcr.io/sentryio/sentry:aa4b37cefca6f83567e7874ccb13612ade00f5b1 ./install.sh

Make sure to keep a back up as this fix is NOT guaranteed to work.

lvdombrkr · October 22, 2020, 10:38am

@BYK we have the same issue. after upgrade from 9.12 -> 20.10.1 no events.
(No events after upgrade to 20.10.1)

Regarding your last answer : “fix needs to run with your old data” , you mean command should be run when we are still on 9.1.2?

Kristaps

mitmafiy · October 22, 2020, 2:15pm

I tried upgrade 9.1.2 to 20.10.1 with your image, but after upgraded I lose all my events( I need to attach some upgrade data logs?

BYK · October 22, 2020, 2:42pm

Might prove useful. We’ll keep working on this, thanks for trying

mitmafiy · October 22, 2020, 3:36pm

@BYK sorry, it is my mistake. all right! I see my events and details. My lab stand is outdated for more than 14 days and std value for filter hid my events. I increased the filter to 90 days and events and details is displayed correctly.

BYK · October 22, 2020, 3:54pm

Oh, this is excellent news thanks for reporting back!

We’ll merge the PR soon then

Topic		Replies	Views
Sentry 10 error after updating Docker images On-Premise	8	4234	March 11, 2020
Upgrading 9.1.2 to 20.* events show "There was an error loading data." On-Premise	1	1356	October 12, 2020
Event 9.1.2 => 20.11.1 migration On-Premise	3	1846	December 11, 2020
Sentry Upgrade from 20.7.2 to 21.6.3 On-Premise	6	1546	October 1, 2021
Error importing events during upgrade from 9.1.2 to 20.11 (and up) On-Premise	13	1916	February 21, 2021

After upgrade from 9.1.2 to 10.0/20.9/latest no events details

Related topics