After upgrade from 9.1.2 to 10.0/20.9/latest no events details

Hi,
I have sentry 9.1.2, deploy in docker from onpromice and i will need uprgade sentry to 20.9.
After upgrade from 9.1.2 to 10.0 or 20.9/latest, I receive the same one error when to open the details of an events:


Now I upgrade sentry 9.1.2 to 10.0.0. After upgrade, all projects, issues and users all right.

But when I try open events details, I have problem described above and see error from Sentry web instances themselves:
изображение


migration log clickhouse:

  • exec gosu snuba snuba migrate
    2020-10-13 14:02:05,957 Migrating dataset outcomes_raw
    2020-10-13 14:02:05,972 Executing migration: ALTER TABLE outcomes_raw_local ADD COLUMN size Nullable(UInt32)
    2020-10-13 14:02:05,977 Column ‘size’ exists in local ClickHouse but not in schema!
    2020-10-13 14:02:05,978 Migrating dataset events
    2020-10-13 14:02:05,986 Migrating dataset groupassignee
    2020-10-13 14:02:05,992 Migrating dataset transactions
    2020-10-13 14:02:05,996 Column ‘transaction_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
    2020-10-13 14:02:05,996 Column ‘transaction_status’ type differs between local ClickHouse and schema! (expected: UInt8 DEFAULT 2, is: UInt8)
    2020-10-13 14:02:05,996 Column ‘_start_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(start_ts), is: Date)
    2020-10-13 14:02:05,996 Column ‘_finish_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(finish_ts), is: Date)
    2020-10-13 14:02:05,996 Column ‘user’ type differs between local ClickHouse and schema! (expected: String DEFAULT ‘’, is: String)
    2020-10-13 14:02:05,996 Column ‘user_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
    2020-10-13 14:02:05,996 Column ‘sdk_name’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
    2020-10-13 14:02:05,996 Column ‘sdk_version’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
    2020-10-13 14:02:05,999 Column ‘transaction_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
    2020-10-13 14:02:05,999 Column ‘transaction_status’ type differs between local ClickHouse and schema! (expected: UInt8 DEFAULT 2, is: UInt8)
    2020-10-13 14:02:05,999 Column ‘_start_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(start_ts), is: Date)
    2020-10-13 14:02:05,999 Column ‘_finish_date’ type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(finish_ts), is: Date)
    2020-10-13 14:02:05,999 Column ‘user’ type differs between local ClickHouse and schema! (expected: String DEFAULT ‘’, is: String)
    2020-10-13 14:02:05,999 Column ‘user_hash’ type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
    2020-10-13 14:02:05,999 Column ‘sdk_name’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
    2020-10-13 14:02:05,999 Column ‘sdk_version’ type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT ‘’, is: LowCardinality(String))
    2020-10-13 14:02:05,999 Migrating dataset groupedmessage
    2020-10-13 14:02:06,005 Migrating dataset discover2020-10-13 14:02:06,006 Migrating dataset outcomes
    2020-10-13 14:02:06,009 Column ‘size’ exists in local ClickHouse but not in schema!

migration log postgresql:

14:02:14 [INFO] sentry.plugins.github: apps-not-configured
Faking migration for contenttypes.0001_initial
Faking migration for contenttypes.0002_remove_content_type_name
Faking migration for auth.0001_initial
Faking migration for auth.0002_alter_permission_name_max_length
Faking migration for auth.0003_alter_user_email_max_length
Faking migration for auth.0004_alter_user_username_opts
Faking migration for auth.0005_alter_user_last_login_null
Faking migration for auth.0006_require_contenttypes_0002
Faking migration for sites.0001_initial
Faking migration for admin.0001_initial
Faking migration for sessions.0001_initial
^[[36;1mOperations to perform:^[[0m
^[[1m Apply all migrations: ^[[0mnodestore, sessions, admin, sentry, jira_ac, sites, auth, social_auth, contenttypes
^[[36;1mRunning migrations:^[[0m
Rendering model states…^[[32;1m DONE^[[0m
Applying admin.0002_logentry_remove_auto_add…^[[32;1m OK^[[0m
Applying auth.0007_alter_validators_add_error_messages…^[[32;1m OK^[[0m
Applying sentry.0002_912_to_recent…^[[32;1m OK^[[0m
Applying sentry.0003_auto_20191022_0122…^[[32;1m OK^[[0m
Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau…^[[32;1m OK^[[0m
Applying sentry.0005_fix_content_types…^[[32;1m OK^[[0m
Applying sentry.0006_sentryapp_date_published…^[[32;1m OK^[[0m
Applying sentry.0007_auto_20191029_0131…^[[32;1m OK^[[0m
Applying sentry.0008_auto_20191030_0016…^[[32;1m OK^[[0m
Applying sentry.0009_auto_20191101_1608…^[[32;1m OK^[[0m
Applying sentry.0010_auto_20191104_1641…^[[32;1m OK^[[0m
Applying sentry.0011_remove_pagerdutyservice_service_id_from_state…^[[32;1m OK^[[0m
Applying sentry.0012_remove_pagerdutyservice_service_id…^[[32;1m OK^[[0m
Applying sentry.0013_auto_20191111_1829…^[[32;1m OK^[[0m
Applying sentry.0014_delete_sentryappwebhookerror…^[[32;1m OK^[[0m
Applying sentry.0015_delete_sentryappwebhookerror_db…^[[32;1m OK^[[0m
Applying sentry.0016_delete_alert_rule_deprecated_fields…^[[32;1m OK^[[0m
Applying sentry.0017_incident_aggregation…^[[32;1m OK^[[0m
Applying sentry.0018_discoversavedquery_version…^[[32;1m OK^[[0m
Applying sentry.0019_auto_20191114_2040…^[[32;1m OK^[[0m
Applying sentry.0020_auto_20191125_1420…^[[32;1m OK^[[0m
Applying sentry.0021_auto_20191203_1803…^[[32;1m OK^[[0m
Applying sentry.0021_auto_20191202_1716…^[[32;1m OK^[[0m Applying sentry.0022_merge…^[[32;1m OK^[[0m
Applying sentry.0023_hide_environment_none_20191126…^[[32;1m OK^[[0m
Applying sentry.0024_auto_20191230_2052…Events to process: 66071
Done

any help would be helpful

2 Likes

This part of the logs suggest that your migrations got interrupted in the middle. Possibly due to an OOM kill. I’d try rerunning the migrations with more RAM available.

Also I strongly recommend trying to upgrade to a more recent version rather than 10.0.0 (like 20.9.0 or the upcoming 20.10.0 which will be released on Thursday) as they have important bugfixes to make this migration easier.

Thanks for your reply.
VM RAM - 12GB.
I tried directly upgrade from 9.1.2 to 20.9 or latest and I have the same problem with the events details display. Then I started trying to upgrade versions sequentially and couldn’t upgrade correctly from 9.1.2 to 10.0.
Okay, I’ll wait 20.10.0 and I will try upgrade to this version.

This may still be exhausted. Are you able to check OOM killer logs?

Yes, I checked logs, but are no OOMk events in the logs.

Now I have postgresql v9.6. Before to upgrade sentry to 20.10, I need upgrade postgresql to 12 or 13?

We still use Postgres 9.6:

Today, I uprgade 9.1.2 to 20.10, but unfortunately the problem remained.

migration log postgresql:

08:11:55 [INFO] sentry.plugins.github: apps-not-configured
Faking migration for contenttypes.0001_initial
Faking migration for contenttypes.0002_remove_content_type_name
Faking migration for auth.0001_initial
Faking migration for auth.0002_alter_permission_name_max_length
Faking migration for auth.0003_alter_user_email_max_length
Faking migration for auth.0004_alter_user_username_opts
Faking migration for auth.0005_alter_user_last_login_null
Faking migration for auth.0006_require_contenttypes_0002
Faking migration for sites.0001_initial
Faking migration for admin.0001_initial
Faking migration for sessions.0001_initial
Operations to perform:
Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying admin.0002_logentry_remove_auto_add… OK
Applying auth.0007_alter_validators_add_error_messages… OK
Applying auth.0008_alter_user_username_max_length… OK
Applying sentry.0002_912_to_recent… OK
Applying sentry.0003_auto_20191022_0122… OK
Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau… OK
Applying sentry.0005_fix_content_types… OK
Applying sentry.0006_sentryapp_date_published… OK
Applying sentry.0007_auto_20191029_0131… OK
Applying sentry.0008_auto_20191030_0016… OK
Applying sentry.0009_auto_20191101_1608… OK
Applying sentry.0010_auto_20191104_1641… OK
Applying sentry.0011_remove_pagerdutyservice_service_id_from_state… OK
Applying sentry.0012_remove_pagerdutyservice_service_id… OK
Applying sentry.0013_auto_20191111_1829… OK
Applying sentry.0014_delete_sentryappwebhookerror… OK
Applying sentry.0015_delete_sentryappwebhookerror_db… OK
Applying sentry.0016_delete_alert_rule_deprecated_fields… OK
Applying sentry.0017_incident_aggregation… OK
Applying sentry.0018_discoversavedquery_version… OK
Applying sentry.0019_auto_20191114_2040… OK
Applying sentry.0020_auto_20191125_1420… OK
Applying sentry.0021_auto_20191203_1803… OK
Applying sentry.0021_auto_20191202_1716… OK
Applying sentry.0022_merge… OK
Applying sentry.0023_hide_environment_none_20191126… OK
Applying sentry.0024_auto_20191230_2052…Events to process: 62444

Event migration done. Migrated 62444 of 62444 events.

OK
Applying sentry.0025_organizationaccessrequest_requester… OK
Applying sentry.0026_delete_event… OK
Applying sentry.0027_exporteddata… OK
Applying sentry.0028_user_reports… OK
Applying sentry.0029_discover_query_upgrade… OK
Applying sentry.0030_auto_20200201_0039… OK
Applying sentry.0031_delete_alert_rules_and_incidents… OK
User Options: 100% |#############################################| Time: 0:00:00
OK
Applying sentry.0033_auto_20200210_2137… OK
Applying sentry.0034_auto_20200210_2311… OK
Applying sentry.0035_auto_20200127_1711… OK
Applying sentry.0036_auto_20200213_0106… OK
Applying sentry.0037_auto_20200213_0140… OK
Applying sentry.0038_auto_20200213_1904… OK
Applying sentry.0039_delete_incidentsuspectcommit… OK
Applying sentry.0040_remove_incidentsuspectcommittable… OK
Applying sentry.0041_incidenttrigger_date_modified… OK
Applying sentry.0042_auto_20200214_1607… OK
Applying sentry.0043_auto_20200218_1903… OK
Applying sentry.0044_auto_20200219_0018… OK
Applying sentry.0045_remove_incidentactivity_event_stats_snapshot… OK
Applying sentry.0046_auto_20200221_1735… OK
Applying sentry.0047_auto_20200224_2319… OK
Applying sentry.0048_auto_20200302_1825… OK
Applying sentry.0049_auto_20200304_0254… OK
Applying sentry.0050_auto_20200306_2346… OK
Audit Log Entrys: N/A% | | ETA: --:–:--None
08:25:59 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
None

08:26:00 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
Audit Log Entrys: 100% |#########################################| Time: 0:00:01
OK
Applying sentry.0052_organizationonboardingtask_completion_seen… OK
Applying sentry.0053_migrate_alert_task_onboarding… OK
Applying sentry.0054_create_key_transaction… OK
Applying sentry.0055_query_subscription_status… OK
Applying sentry.0056_remove_old_functions… OK
Applying sentry.0057_remove_unused_project_flag… OK
Applying sentry.0058_project_issue_alerts_targeting… OK
Applying sentry.0059_add_new_sentry_app_features… OK
Applying sentry.0060_add_file_eventattachment_index… OK
Applying sentry.0061_alertrule_partial_index… OK
Applying sentry.0062_key_transactions_unique_with_owner… OK
Applying sentry.0063_drop_alertrule_constraint… OK
Applying sentry.0064_project_has_transactions… OK
Applying sentry.0065_add_incident_status_method… OK
Applying sentry.0066_alertrule_manager… OK
Organizations: 100% |############################################| Time: 0:00:00
OK
Applying sentry.0068_project_default_flags… OK
Applying sentry.0069_remove_tracked_superusers… OK
Applying sentry.0070_incident_snapshot_support… OK
Applying sentry.0071_add_default_fields_model_subclass… OK
Applying sentry.0072_alert_rules_query_changes… OK
Applying sentry.0073_migrate_alert_query_model… OK
Applying sentry.0074_add_metric_alert_feature… OK
Applying sentry.0075_metric_alerts_fix_releases… OK
Applying sentry.0076_alert_rules_disable_constraints… OK
Applying sentry.0077_alert_query_col_drop_state… OK
Applying sentry.0078_incident_field_updates… OK
Applying sentry.0079_incidents_remove_query_field_state… OK
Applying sentry.0080_alert_rules_drop_unused_tables_cols… OK
Applying sentry.0081_add_integraiton_upgrade_audit_log… OK
Applying sentry.0082_alert_rules_threshold_float… OK
Applying sentry.0083_add_max_length_webhook_url… OK
Applying sentry.0084_exported_data_blobs… OK
Applying sentry.0085_fix_error_rate_snuba_query… OK
Applying sentry.0086_sentry_app_installation_for_provider… OK
Applying sentry.0087_fix_time_series_data_type… OK
Applying sentry.0088_rule_level_resolve_threshold_type… OK
Applying sentry.0089_rule_level_fields_backfill… OK
Audit Log Entrys: N/A% | | ETA: --:–:--None
08:26:16 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
None

08:26:17 [ERROR] sentry.runner.initializer: django.compat.model-unpickle-compat (model_id=(‘sentry’, ‘Team’) factory=<function __simple_class_factory_compat at 0x7f6ddcdd9de8> attrs=[])
Audit Log Entrys: 100% |#########################################| Time: 0:00:01
OK
Applying sentry.0091_alertruleactivity… OK
Applying sentry.0092_remove_trigger_threshold_type_nullable… OK
Applying sentry.0093_make_identity_user_id_textfield… OK
Applying sentry.0094_cleanup_unreferenced_event_files… OK
Applying sentry.0095_ruleactivity… OK
Applying sentry.0096_sentry_app_component_skip_load_on_open… OK
Applying sentry.0097_add_sentry_app_id_to_sentry_alertruletriggeraction… OK
Applying sentry.0098_add-performance-onboarding… OK
Projects: 100% |#################################################| Time: 0:00:00
OK
Applying sentry.0100_file_type_on_event_attachment… OK
Applying sentry.0101_backfill_file_type_on_event_attachment… OK
Applying sentry.0102_collect_relay_analytics… OK
Applying sentry.0103_project_has_alert_filters… OK
Applying sentry.0104_collect_relay_public_key_usage… OK
Applying sentry.0105_remove_nullability_of_event_attachment_type… OK
Applying sentry.0106_service_hook_project_id_nullable… OK
Applying sentry.0107_remove_spaces_from_slugs… OK
Applying sentry.0108_update_fileblob_action… OK
Applying sentry.0109_sentry_app_creator… OK
Applying sentry.0110_sentry_app_creator_backill… OK
Applying sentry.0111_snuba_query_event_type… OK
Applying sentry.0112_groupinboxmodel… OK
Applying sentry.0113_add_repositoryprojectpathconfig… OK
Applying sites.0002_alter_domain_unique… OK

error when i opens events details:



error from Sentry web instances themselves:

docker-compouse log:

nginx_1 | 192.168.100.65 - - [16/Oct/2020:10:27:40 +0000] “GET /api/0/issues/2464/ HTTP/1.1” 200 3317 “http://192.168.100.74:9000/organizations/sentry/issues/2464/?project=17&query=is%3Aunresolved” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0” “-”
web_1 | 10:27:40 [WARNING] django.request: Not Found: /api/0/issues/2464/events/latest/ (status_code=404 request=<WSGIRequest: GET u’/api/0/issues/2464/events/latest/’>)
nginx_1 | 192.168.100.65 - - [16/Oct/2020:10:27:40 +0000] “GET /api/0/issues/2464/events/latest/ HTTP/1.1” 404 28 “http://192.168.100.74:9000/organizations/sentry/issues/2464/?project=17&query=is%3Aunresolved” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0” “-”
nginx_1 | 2020/10/16 10:27:40 [warn] 6#6: *274 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000015, client: 172.21.0.1, server: , request: “POST /api/1/store/?sentry_key=a4fd59524c223ce27b701&sentry_version=7 HTTP/1.0”, host: “http://192.168.100.74:9000/
nginx_1 | 172.21.0.1 - - [16/Oct/2020:10:27:40 +0000] “POST /api/1/store/?sentry_key=a4fd59524c223ce27b701&sentry_version=7 HTTP/1.0” 200 41 “http://192.168.100.74:9000/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0” “192.168.100.65”
ingest-consumer_1 | 10:27:41 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u’ingest-events’, 0): [18L, 18L]}): forced:False size:False time:True
ingest-consumer_1 | 10:27:41 [INFO] batching-kafka-consumer: Worker flush took 5ms
nginx_1 | 172.21.0.1 - - [16/Oct/2020:10:27:42 +0000] “GET /_static/1602843379/sentry/dist/vendor.js.map HTTP/1.1” 200 1661333 “-” “sentry/20.10.0 (https://sentry.io)” “-”
snuba-transactions-consumer_1 | 2020-10-16 10:27:43,532 Completed processing <Batch: 1 message, open for 1.00 seconds>.
snuba-outcomes-consumer_1 | 2020-10-16 10:27:43,539 Completed processing <Batch: 1 message, open for 1.01 seconds>.
snuba-consumer_1 | 2020-10-16 10:27:43,549 Completed processing <Batch: 1 message, open for 1.02 seconds>.

I looked in table “nodestore_node”:

          id                

----------------------------------±------------------------------
20b661d14d9ea95ebb3c606ca84a1281
2c1b220825cda454a73937368ce71fb8
1f29041a7ff0932ec1c85c68313f2fb0
87b0b320a2238c0049b92a09dab0ad52
ac64e478e332eabe9ddb3908d929c2e9
AyVG1I5aQcSoBK0V8iLukQ==
He/BHxcfSHixscnM9xGJ8g==
qPIBpNj0QpSa9oCwPY9ixg==
eKXnBhfQTVeHPVkNhtcZmA==

format id before upgrade: AyVG1I5aQcSoBK0V8iLukQ==
format id after upgrade: 20b661d14d9ea95ebb3c606ca84a1281

perhaps that’s the problem?

1 Like

apparently, this ID changed when I updated my sentry 8.15 to 9.1.2. and on v9.1.2, I can see that my events/issues are shown correctly.
when I updated 9.1.2 to 20.x, I see that query to detailes is executed using id with the UUID format. But old events detailes has old format id.

how do you think this can be fixed?

1 Like

How old are these events?

they’re not old. I just week ago upgraded sentry to 9.1.2. Our team long worked on v8.15)

Ah, sorry I misunderstood. Okay investigating this still.

@mitmafiy - thanks a ton for the detailed information you have shared with us! I think you have helped us identify a long-standing migration issue. We are currently actively investigating this and will be reporting back here.

2 Likes

@mitmafiy we have a potential fix in place but the fix needs to run with your old data. If you can do it, please run the following in the 20.10.1 checkout of the onpremise repo:

SENTRY_IMAGE=us.gcr.io/sentryio/sentry:aa4b37cefca6f83567e7874ccb13612ade00f5b1 ./install.sh

Make sure to keep a back up as this fix is NOT guaranteed to work. :slight_smile:

1 Like

@BYK we have the same issue. after upgrade from 9.12 -> 20.10.1 no events.
(No events after upgrade to 20.10.1)

Regarding your last answer : “fix needs to run with your old data” , you mean command should be run when we are still on 9.1.2?

Kristaps

I tried upgrade 9.1.2 to 20.10.1 with your image, but after upgraded I lose all my events( I need to attach some upgrade data logs?

Might prove useful. We’ll keep working on this, thanks for trying :slight_smile:

@BYK sorry, it is my mistake. all right! I see my events and details. My lab stand is outdated for more than 14 days and std value for filter hid my events. I increased the filter to 90 days and events and details is displayed correctly.

1 Like

Oh, this is excellent news thanks for reporting back!

We’ll merge the PR soon then :slight_smile:

1 Like