I write lots of services and client apps -- most of them as proof-of-concept or experimental implementations to test out design ideas and/or explore technical details. One of the things I learned early on is how frustrating (and dangerous) HTTP POST can be. And after years of wrestling with various approaches and workarounds, I finally gave up and just stopped using HTTP POST altogether.
Yep, you heard me. I don't use HTTP POST anymore -- and here's my story.
Anyone who has created HTTP services has experienced some form of the Lost Response problem. You send a request to the server and nothing comes back. I don't mean the response is empty, I mean nothing comes back. Zero. Nada. Zip. Zilch.
Now, if you are sending an HTTP GET request and it fails to respond, there is no real harm done. You can safely try again -- as many times as you wish. In fact, a handful of HTTP libraries have this repeatable GET built into the module. Failed or missing responses to GET result in a random millisecond pause and another attempt (up to some pre-set maximum). Cool.
This also works for HTTP PUT and HTTP DELETE. If you get a failed or missing response, you can confidently repeat the action without creating any additional problems. That is unless you make so many attempts you flip the server's "denial of service" alarm, and then you have another problem on your hands. I haven't found a library that automatically retries these (as in the GET case) but I wish they did.
But (and you see where I'm going with this), you can't employ this "repeated attempts" pattern when using HTTP POST. Why? Because idempotency, that's why.
HTTP wisely built into the protocol two ways to characterise HTTP methods. They can be "safe" (e.g. GET, HEAD) or "unsafe" (PUT, POST, DELETE, PATCH). And they can be "idempotent" (GET, HEAD, PUT, DELETE) or "non-idempotent" (POST, PATCH). For those who are curious, there is a nice IANA registry of HTTP methods that lists their safety & idempotency values.
An idempotent method is one that returns the same result no matter how many times you execute it. Think about the HTTP command DELETE /persons/123. No matter how many times you send this request to a server, there is only going to be one record removed from the system -- the one associated with the URL /persons/123. That's idempotency.
Now try this with HTTP POST /persons/ and a JSON body of {name=Mike}. Do this a few times and you get a handful of records in your system that have the same value for name. All fine if that's what you want. But not so fine for many cases.
And that gets us back to the Lost Response problem. Consider the case where you send an HTTP POST to /persons/ with {name=Mike} and you get no response back at all. No Status 201. No Status 400. No Status 500. Just nothing.
Now what?
Do you assume the request never got there and just repeat it? Or do you assume the request was successful and that the network somehow gobbled up the server's "201 Created" response? Maybe what you do is make an HTTP GET /persons/ request to see if your Mike record has been created and, if not, make that create request again.
Kinda fiddly. And I've done all these of course. As I suspect most of you have, too. As humans, this is a pain, but not a disaster. It happens. Meh.
But the context changes when it is not a human making the request. A human can do some reasoning and sort things out. Bots? Not so much. Bots, or any machine-to-machine (M2M) interactions, really need a consistent, easy to grok set of use cases. And, in my experience, HTTP POST is not one of them.
M2M cases are examples where we definitely need an idempotent way to create new resources over HTTP. And it's been done in the past. More than once.
There have been a few serious attempts to make HTTP POST repeatable. HTTPLR was one of them. It operates on the notion of creating an "exchange URL" and then working through several steps. Another attempt is the POE or Post Once Exactly pattern. POE uses special headers to track and identify cases where POST cannot be repeated. They both do what they advertise, but for whatever reason these ideas haven't caught on. I suspect they are too involved to be commonly used.
Both HTTPLR and POE are attempts to treat POST as if it was idempotent (or at least make up for that fact it is not). But I've been using a much more direct approach. One that just bypasses POST altogether. And it works just fine for me.
Years ago, long after HTTP was created, it was "decided" that people should map common database actions onto HTTP methods. Not a great idea, but one that really took hold. So, for more than a decade, HTTP programmers have used POST to create, GET to read, PUT to update, and DELETE to delete -- hence the CRUD label. There were other similar ideas in contention like BREAD, DAVE, and (my favorite) CRAP. But CRUD won out.
Personally though, I ignore the standard CRUD mapping and just use PUT. And before you get excited -- yes, I also use PUT to update an existing resource. So, how do I make sure the intent of one PUT request is interpreted as "create-this-new-resource" and another PUT request is interpreted as "update-this-existing-resource”? It's easy.
I do it with two HTTP headers that have been in the HTTP specification since 1997.
When you send an HTTP PUT request with a body (like {name=Mike}), you can also send a header telling HTTP the PUT is conditional. In other words, the server needs to confirm another server-side condition before agreeing to do what the client application requested.
When you want to create a new resource using PUT you just need to send the following:
**** REQUEST ****
PUT /persons/123
Host: api.example.org
Content-Type: application/json
Content-Length: XXX
If-None-Match : "*"
{"name":"Mike"}
**** RESPONSE ****
201 Created
Location: http://api.example.org/persons/123
Note the If-None-Match header. That's the magic. It tells the server that the PUT should only be completed if there are no existing resources at the supplied URL (/persons/123). The typical response (201 Created) includes a Location header pointing to the newly created resource. And you can view that with an HTTP GET.
**** REQUEST ****
GET /persons/123
Host: api.example.org
Accept: application/json
**** RESPONSE ****
200 OK Content-Type: application/json
Content-Length: XXX
ETag: "1q2w3e4r5t6yz"
{"name":"Mike"}
Note the ETag header in the response. This is a special hash value that is unique for each version of a resource. Each time a resource gets updated, it gets a new ETag. We'll need this when we try to update the resource.
And, as you expect, you can still use HTTP PUT to update an existing resource. That's done by passing the ETag for the resource back to the server in the If-Match conditional header. Like this:
**** REQUEST ****
PUT /persons/123
Host: api.example.org
Content-Type: application/json
Content-Length: XXX
If-Match : "1q2w3e4r5t6yz"
{"name":"Mary"}
**** RESPONSE ****
200 OK
Content-Type: application/json
Content-Length: XXX
ETag: "zaxscdvfbgnh"
{"name":"Mary"}
With the PUT-CREATE pattern in place, I can safely repeat a resource "create" request even if I encounter the Lost Response problem. I can do this because I am using PUT, which is idempotent, along with a conditional header (If-None-Match for create and If-Match for update). And all this has been built into HTTP for almost a quarter of a century. No need for any fancy new headers or special resource exchanges. Just basic HTTP.
With a simple, straight-forward idempotent creation pattern over HTTP it is now possible to confidently code M2M create requests. Adopting this approach has made my life a whole lot easier when programming both HTTP servers and HTTP clients. Yes, I had to tweak my stock client code to burp out the new "PUT, If-None-Match" requests for creation. I also had to tweak my server code to consistently emit ETag values and pay attention to If-None-Match and If-Match headers. But that was really trivial.
And the results have been really quite good.
This is especially true when I'm experimenting with crafting HTTP create requests when I don't quite know the proper input rules, or I am playing with other minor details in the creation requests. I can be confident I won't mistakenly overwrite an existing resource and that any failure or Lost Response cases are no longer a "blocker" that needs special attention.
In fact, I now find myself frustrated that many of the external APIs I deal with still don't support this clearly superior pattern for creating HTTP resources. I'd love to see this PUT-CREATE approach become much more widespread and am hoping that reading this article will encourage you to do your part to add support for PUT-CREATE to your services and client apps.
And for those who do, thanks for making creating HTTP resources more repeatable and reliable!