When implementing API clients, you tend to always run in to the situation where you need to test if the client you’ve tirelessly worked on actually works. All your unit-tests are passing; now you just have to make sure! So you loosely wire it up in a console application and send a request, to then dump the response to a file or similar — a “manual unit test”.
I would argue that this is in fact the most important test of all, as it is the only test that truly verifies whether your client is working. Why leave it to manual testing? i.e.; wouldn’t a unit-test be nice for this as well? I would sleep better if Travis reported that Sentry accepted my request before every deploy.
So the idea in short is: Send a JSON-representation of an event to sentry, then have the API parse it and then respond accordingly, the only difference being that this event would not get saved.
I think the most elegant solution for this would be to simply provide a sandbox DSN, say:
Or to have a public sandbox DSN for this purpose, say:
http://sandbox@sentry.io/1
The sandbox would follow all the semantics of the live environment, only it would not save an event, and always reply with an “agreed upon” event ID. As an extension it would be nice if the API responded with the “would be saved”-event, that way validation could be made more thoroughly on the client-side assertions.
I do realize this is somewhat of a “luxury feature”, but it is not uncommon for API’s to provide sandboxes.
I think what we’ll end up doing here is providing a JSON schema to validate against. I agree that sometimes you just want that end to end test, but a JSON schema is probably the closest it can get without actually testing over the wire. I am -1 on having an endpoint in our production that can be hit purely for unit testing. It’s trivial enough to create a project for manual testing against our live servers, but I don’t want to encourage automated testing against us. I’d rather us provide a JSON schema to test against, and the transport is then implementation detail.
I think what we’ll end up doing here is providing a JSON schema to validate against.
I think that’s a great idea! I also particularly like it because it keeps everything local.
The point here is essentially to ensure that the JSON generated on my end, will be accepted when posted. So as long as the JSON schema is accurate in that regard, it fulfills this purpose.
Verifying a JSON blob against a schema, is also rather easy.
It misses the mark in some regards though; for instance, if I were to use the wrong character encoding for my request, albeit it being a different test entirely.
It’s trivial enough to create a project for manual testing against our live servers, but I don’t want to encourage automated testing against us.
As is my standpoint, that is partly why I put forth this proposition. However; a JSON-schema is a great improvement upon the initial idea, as it reduces the bytes moved both across the wire and the Sentry infrastructure to zero.
I’d rather us provide a JSON schema to test against, and the transport is then implementation detail.
I also think this is the best solution for the particular problem I’m trying to solve, thanks for a good interpretation of it, and a great solution!
Are the domain restrictions enforced upon the properties of these objects defined somewhere already? If so maybe a schema could be tooled/generated from existing source code with little effort.
Will the schema be posted to the docs when/if available?
Our current validation implementation is really messy, so no automated tool would be able to generate it. But I can say that new stuff is getting JSON schemas implemented to go with them as the base for validation.
As for when this will bleed into the main event endpoint… unsure yet and we haven’t planned it. But we’re working towards it. And yeah, we’d definitely document this once we’re done. Might even make sense to expose them over HTTP so you can fetch the schema from us on demand. But this hasn’t been discussed at all internally.
