Sentry's APM Docs [Alpha]

SDKs
1

THIS POST IS OUTDATED, please visit https://docs.sentry.io/performance/distributed-tracing/

Outdated

Sentry’s APM (Application Performance Management) features provide users with a view of the general health of their system. The APM tools will give insight on an application’s performance bottleneck, even when there aren’t errors.

Our initial rollout of APM includes our Python SDK. If you have version >= 0.14.2 the APM features are available to you.

Getting started – Python

Sending Traces

To send any traces, set the traces_sample_rate
to a nonzero value. The following configuration will capture 10% of
all your transactions:

import sentry_sdk
    
sentry_sdk.init("___PUBLIC_DSN___", traces_sample_rate=0.1)

Automating Traces

Many integrations for popular frameworks automatically capture traces. If you already have any of the following frameworks set up for error reporting, you will start to see traces immediately:

  • All WSGI-based frameworks (Django, Flask, Pyramid, Falcon, Bottle)
  • Celery
  • Redis Queue (RQ)

We are creating spans for the following operations within a transaction:

  • Database that uses SQLAlchemy or the Django ORM
  • HTTP requests made with requests or the stdlib
  • Spawned subprocesses

If you want to automatically enable all relevant transactions, you can do:

import sentry_sdk

sentry_sdk.init("___PUBLIC_DSN___", _experiments={"auto_enabling_integrations": True})

Manual Tracing

Managing Transactions

Let’s say you want to create a transaction for an expensive operation process_item and
send the result to Sentry:

import sentry_sdk

while True:
     item = get_from_queue()

     with sentry_sdk.start_span(op="task", transaction=item.get_transaction()):
         # process_item may create more spans internally (see next examples)
         process_item(item)

Adding additional Spans to the transaction

Consider the next example is called somewhere in the process_item function from before. Our SDK can determine if there is a current open transaction and add all newly created spans as child operations to the transaction. Keep in mind; each individual span also needs to be finished; otherwise, it will not show up in the transaction.

import sentry_sdk

with sentry_sdk.start_span(op="http", description="GET /") as span:
    response = my_custom_http_library.request("GET", "/")
    span.set_tag("http.status_code", response.status_code)
    span.set_data("http.foobarsessionid", get_foobar_sessionid())

The value of op and description can be freely chosen.

Getting started – JavaScript Browser

To use our new APM features, you need to use a new beta release.

$ npm install @sentry/browser@5.10.0-rc.0
$ npm install @sentry/apm@5.10.0-rc.0

Sending Traces

You need to use our new Integration called Tracing that resides in the @sentry/apm package.
To use it simply add it to you Sentry.init call:

import * as Sentry from '@sentry/browser';
import { Integrations as ApmIntegrations } from '@sentry/apm';

Sentry.init({
  dsn: '___PUBLIC_DSN___',
  integrations: [
    new ApmIntegrations.Tracing(),
  ],
});

You can pass a lot of different options to the Tracing Integration see (TracingOptions) but it comes with reasonable defaults out of the box.
You will get a transaction for every page load + all XHR/fetch requests as Spans.

(Preferred) Using Tracing integration for manual Tracing

The Tracing Integration will create a transaction on page load by default, all spans that are started will be attached to it. Also, the Integration will finish the transaction after the default timeout of 500ms of inactivity. Inactivity is if there are no pending XHR/fetch requests. If you want to extend it by adding spans to the transaction you can do it like this, here is an example of how you could profile a React component.

// This line starts an activity (and creates a span).
// As long as you don't pop the activity, the transaction will not be finished and therefore not sent to Sentry.
// If you do not want to create a span out of an activity, just don't provide the second arg.
const activity = ApmIntegrations.Tracing.pushActivity(displayName, {
        data: {},
        op: 'react',
        description: `<${displayName}>`,
});

// Sometime later ...
// When we pop the activity, the Integration will finish the span and after the timeout finish the transaction and send it to Sentry
Integrations.ApmIntegrations.popActivity(activity);

Manual Tracing

Managing Transactions

Let’s say you want to create a transaction for an expensive operation processItem and
send the result to Sentry:

  const transaction = Sentry.getCurrentHub().startSpan({ op: "task",  transaction: item.getTransaction() })
  # processItem may create more spans internally (see next examples)
  processItem(item).then(() => {
    transaction.finish()
  })

Adding additional Spans to the transaction

Consider the next example is called somewhere in the processItem function from before. Our SDK can determine if there is a current open transaction and add all newly created spans as child operations to the transaction. Keep in mind that each individual span also needs to be finished, otherwise it will not show up in the transaction.

function processItem(item, transaction) {
  const span = transaction.child({ op: "http", description: "GET /" })

  return new Promise((resolve, reject) => {
    http.get(`/items/${item.id}`, (response) => {
      response.on('data', () => {});
      response.on('end', () => {
        span.setTag("http.status_code", response.statusCode);
        span.setData("http.foobarsessionid", getFoobarSessionid(response));
        span.finish();
        resolve(response);
      });
    });
  });
}

Getting started – Node.js

To use our new APM features, you need to use a new beta release.

$ npm install @sentry/node@5.10.0-rc.0
$ npm install @sentry/apm@5.10.0-rc.0

Sending Traces

To send any traces, set the tracesSampleRate
to a nonzero value. The following configuration will capture 10% of
all your transactions:

const Sentry = require("@sentry/node");
const Apm = require("@sentry/apm"); // This is required since it patches functions on the hub

Sentry.init({
  dsn: "___PUBLIC_DSN___",
  tracesSampleRate: 0.1
});

Automating Traces

As of Today (Nov 4th 19’), it’s possible to add tracing to all popular frameworks, however, we provide pre-written handlers only for Express.js.

const Sentry = require("@sentry/node");
const express = require("express");
const app = express();

// RequestHandler creates a separate execution-context using domains, so that every transaction/span/breadcrumb has it's own Hub to be attached to
app.use(Sentry.Handlers.requestHandler());
// TracingHandler creates a tracing for every incoming request
app.use(Sentry.Handlers.tracingHandler());

// the rest of your app

app.use(Sentry.Handlers.errorHandler());
app.listen(3000);

We are creating spans for the following operations within a transaction:

  • HTTP requests made with request or get calls using native http and https modules
  • Middlewares (Express.js only)

To enable them respectively:

$ npm install @sentry/integrations@5.8.0-beta.0

const Sentry = require("@sentry/node");
const Integrations = require("@sentry/integrations");
const express = require("express");
const app = express();

Sentry.init({
  dsn: "___PUBLIC_DSN___",
  tracesSampleRate: 0.1,
  integrations: [
    // enable HTTP calls tracing
    new Sentry.Integrations.Http({ tracing: true }),
    // enable Express.js middleware tracing
    new Integrations.Express({ app })
  ],
});

Manual Tracing

Managing Transactions

Let’s say you want to create a transaction for an expensive operation processItem and
send the result to Sentry:

app.use(function processItems(req, res, next) {
  const item = getFromQueue();
  const transaction = Sentry.getCurrentHub().startSpan({ op: "task",  transaction: item.getTransaction() })
  
  # processItem may create more spans internally (see next examples)
  processItem(item, transaction).then(() => {
    transaction.finish();
    next();
  })
});

Adding additional Spans to the transaction

Consider the next example is called somewhere in the processItem function from before. Our SDK can determine if there is a currently open transaction and add all newly created spans as child operations to the transaction. Keep in mind that each individual span also needs to be finished, otherwise it will not show up in the transaction.

function processItem(item, transaction) {
  const span = transaction.child({ op: "http", description: "GET /" })

  return new Promise((resolve, reject) => {
    http.get(`/items/${item.id}`, (response) => {
      response.on('data', () => {});
      response.on('end', () => {
        span.setTag("http.status_code", response.statusCode);
        span.setData("http.foobarsessionid", getFoobarSessionid(response));
        span.finish();
        resolve(response);
      });
    });
  });
}

The value of op and description can be freely chosen.

If you’re unfamiliar with the concepts and terms associated with APM, feel free to take a look at our:

Glossary

Span

The span is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system.

It can be the processing of an HTTP request by a server, the querying of a database or an HTTP call to an external service.

An example of a span that describes a call to the external service:


    {
      "trace_id": "a0fa8803753e40fd8124b21eeb2986b5",
      "parent_span_id": "9c2a6db8c79068a2",
      "span_id": "8c931f4740435fb8",
      "start_timestamp": 1563890702.134,
      "same_process_as_parent": true,
      "description": "http://httpbin.org/base64/aGVsbG8gd29ybGQK GET",
      "tags": { "http.status_code": 200, "error": false },
      "timestamp": "1563890702.934",
      "op": "http",
      "data": {
        "url": "http://httpbin.org/base64/aGVsbG8gd29ybGQK",
        "status_code": 200,
        "reason": "OK",
        "method": "GET"
      }
    }

Properties

Each span has a unique identifier kept in the span_id attribute.

Start time and end time are stored in the start_timestamp and timestamp. (End time is captured in the attribute timestamp and not the end_timestamp for now because of compatibility reasons.)

The name of the operation is stored in the “op” parameter. Examples are http, sql.query, redis.command.

Additional data about the operation can be stored in the data and tags attributes.

Hierarchy

Each span can be connected to other spans using a parent-child hierarchy. (There is also a “Follows from” relationship that will be explained later on.)

For example, a span called updateJiraTask would have child spans like Check permissions, Update Task, sql.update, send.webhooks.

Each sub-task would have an id in the attribute parent_span_id id of the “updateTask” span.

Transaction

There already was a concept of transaction in the Sentry SDKs before we started with APM.

Transaction is an event attribute that provides initial context to the error report. This makes it possible for the error to be linked to a specific part of the system. An example of the transaction name can be “/users//” which is the name for the endpoint that received the request that failed during the processing.

With APM, the transaction is a parent span to all the spans that are created during the processing of the request in the single server.

Once the request is processed, the transaction is sent together with all child spans to Sentry.

Event Types

The SDKs already collect a lot of information about the processing of the request. In the case of an error, the SDK attaches the info to the error report to provide rich context of what happened before the error appeared. (Information like tags and breadcrumbs).

If the error doesn’t appear during the processing of the request, we discard the context data once processing is complete.

We want to leverage this context collection to build spans, and once a request is finished, we want to send the data to Sentry.

Example of the transaction event type:

    {
      "start_timestamp": "2019-06-14T14:01:40Z",     
      "transaction": "tracing.decode_base64",      
      "type": "transaction",                       
      "event_id": "2975518984734ef49d2f75db4e928ddc",
      "contexts": {
    	"trace": {                   
    		"parent_span_id": "946edde6ee421874",      
    		"trace_id": "a0fa8803753e40fd8124b21eeb2986b5", 
    		"span_id": "9c2a6db8c79068a2"   
    	}
      },
      "timestamp": "2019-06-14T14:01:41Z",    
      "server_name": "apfeltasche.local",
      "extra": {
    	"sys.argv": ["/Users/untitaker/projects/sentry-python/.venv/bin/flask","worker"]
      },
      "modules": {
    	"more-itertools": "5.0.0",
    	"six": "1.12.0",
    	"funcsigs": "1.0.2"
      },
      "platform": "python",
      "spans": [
        {
          "op": "db",
          "description": "SELECT * from countries where id = ?",
          "start_timestamp": 1562681591,
          "timestamp": 1562681591,
          "parent_span_id": "9c2a6db8c79068a2",
          "trace_id": "a0fa8803753e40fd8124b21eeb2986b5",
          "data": {},
          "tags": {}
        }
      ],
      "breadcrumbs": [
    	  {
    		"category": "httplib",
    	  	"data": {
    		  	"url": "http://httpbin.org/base64/aGVsbG8gd29ybGQK",
    		  	"status_code": 200,
    	  		"reason": "OK",
    	  		"method": "GET"
    		  },
    		  "type": "http",
    		  "timestamp": "2019-06-14T12:01:41Z"
    	  }
      ],
      "sdk": {
    	"version": "0.9.0",
    	"name": "sentry.python",
    	"packages": [{ "version": "0.9.0", "name": "pypi:sentry-sdk" }],
    	"integrations": ["argv"]
      }
    }

Trace

A trace is a collection of spans that have the same trace_id attribute. Traces consist of all the spans that were generated by the distributed system during the processing of the input data. Traces are not explicitly started or finished. The start and end of a trace is defined by the transactions and spans participating in the trace.

Propagation

Processing of the input data in the distributed environment means that an arbitrary number of the systems/microservices can be involved.

To connect all the spans that were created during the processing of the input data, we need to be able to propagate the basic identifier. By forwarding trace_id and parent_span_id as data travels through various applications in a microservice ecosystem, we can reconstruct data flows later.

Trace metadata is propagated through http headers that are included in the http request to the external system.

Performance Metric Definitions

As part of APM and transaction data, we’re exposing a few new aggregate metrics to help customers better understand the performance of their applications.

RPM

Requests Per Minute is a way to measure throughput. It is the average of all requests bucketed by the minute for the current time window and query string.

Duration Percentiles

The average, 75% and 95% duration, help guide customers in identifying transactions that are slower than the user’s target SLAs.

Our APM data is sampled so the percentiles we present only represent the data we have. Because of sampling, data ranges, filters, or a low traffic transaction, we will often have a case where data is directionally correct, but not accurate. For example, the average of one number is that number, but that’s not usually what people expect to see.

This inability to be usefully accurate will happen more often in some columns than others. We can calculate an average with less data than a 95th percentile, but it’ll also vary by row. An index page will always get more traffic than /settings/country/belgium/tax.

To signal this lack of accuracy to the user, we’ll show a warning to users:

  • If there are fewer than 5 data-points, we’ll show “n/a” in the average column.
  • If there are fewer than 20 data-points, we’ll show “n/a” in the 75th percentile
  • If there are fewer than 100 data-points, we’ll show “n/a” in the 95th percentile

The UI will highlight any warnings visually and let the user know that they should increase their sampling rate or make a less precise query.

In addition to warnings, some fields are meaningless at certain cardinalities. For example, if you have one data-point, the 95th percentile means nothing.

  • If the RPM <1, we’ll show “<1”
  • If there are less than 4 data-points, we’ll show “n/a” in the 75th percentile
  • If there are less than 20 data-points, we’ll show “n/a” in the 95th percentile

Percentage Query

This column shows the % of the entire query that this row represents, so you filter your results with the query and then see how much of the result set a particular transaction represents.

2

@HazAT How is this shown in the Web UI. Are there any screenshots? Does this enhance traces or is it only for monitoring when things don’t go wrong?

3

So here is how it looks like right now (two example transactions):

image
image.png1663×1293 463 KB

image
image.png1667×731 127 KB

You can apply to get access to it right here:

Just to let you know, access to this right is very limited since we want to make sure it’s ready for prime time.

4

@HazAT When does APM feature open to the public? And then, Is it possible to use it on on-premise environment?

5

Soon, I’ve updated the inital post, it’s already available in some limited form.

6

https://docs.sentry.io/performance/?platform=javascript mentions installing the @sentry/tracing package from NPM, but I don’t see a package with that name. Did that package get renamed @sentry/apm?

7

Fixed, thx

1 Like