Tag Archives: Upsert

Dataverse Web API Tip #8: Those Mysterious ETags (Conditional Operations)

• , ,

Ever notice the mysterious @odata.etag values that seems to be returned for every entity you GET? What is an ETag, and what can you do with it?

GET {{webApiUrl}}contacts(b8d3f910-1896-eb11-b1ac-000d3a3ac80d)?$select=firstname
(include the "always include" headers)

Response:
{
  "@odata.context": "{{webApiUrl}}$metadata#contacts(firstname)/$entity",
  "@odata.etag": "W/\"842270\"",
  "firstname": "Yvonne",
  "contactid": "b8d3f910-1896-eb11-b1ac-000d3a3ac80d"
}

Think of an ETag (short for “entity tag”) as a record’s version identifier. In the Dataverse world, if you’re fetching the same resource and get back the same ETag, you know that the resource is semantically equivalent to what you previously fetched.

So, yesterday if you did a GET {{webApiUrl}}contacts(b8d3f910-1896-eb11-b1ac-000d3a3ac80d) and the data you received was accompanied with an ETag of W/"1377902", then today you fetch the same resource and get back the same ETag, you know that the resource hasn’t changed.

While there may be some usefulness in performing full entity fetches and then comparing ETags client-side, it’s the If-Match and If-None-Match headers that make ETags shine.

Conditional GET

Let’s go back to our initial example. Instead of doing a straight second fetch today then comparing ETags client-side to determine whether the entity changed, you could add an If-None-Match header to today’s request, setting it to the ETag received yesterday:

GET {{webapiurl}}contacts(b8d3f910-1896-eb11-b1ac-000d3a3ac80d)
If-None-Match: W/"1377902"
(include the other "always include" headers)

On the server, if the entity’s ETag is different from the specified value, the entity will be returned. However, if the ETag you provided matches the entity’s current ETag, a 304 Not Modified response is returned. This signals you that you already have the latest on this entity. At the same time, it saves the cost of having the server send you another copy of that data.

By setting the If-None-Match header, you pushed ETag comparison to the server. This is an improvement over handling ETag comparison client-side—but things keep getting even better….

Conditional Update

Imagine you’re building an API client which fetches an entity’s data then displays it in a UI where it can be edited. What should happen if the entity is updated in Dataverse between when your client retrieves the entity’s details and when it saves changes back to Dataverse? Let’s say a user of your client pulls up an entity Friday before leaving for the weekend. On Monday, they come back in and edit the entity’s details without refreshing (so they’re working with the entity as it looked Friday). Over the weekend, someone else modified the same entity in Dataverse. What should happen when your client’s user clicks “save”: Should their change override the other edit? Should the save error because of the other edit?

In the data world, the second choice (erroring if someone else changed the entity) is known as optimistic concurrency control. In the Dataverse world, optimistic concurrency for an update is easy to implement thanks to the If-Match header.

To perform an update with optimistic concurrency, send the PATCH request with header If-Match set to the ETag the edit is based on (i.e. the ETag received when your API client performed its fetch). Dataverse will allow the update to succeed if the entity’s current ETag matches the specified value. Otherwise, it will error with a 412 Precondition Failed response.

PATCH {{webApiUrl}}contacts(b8d3f910-1896-eb11-b1ac-000d3a3ac80d)
If-Match: W/"1377902"
(include the other "always include" headers)
	
{ … JSON with column edits goes here … }

This ensures you that the edit will only be accepted if the entity wasn’t changed on the server during the intervening time. Nice!

Conditional Deletes

If-Match can be very important with DELETEs. How often do you want a deletion to succeed if someone edited the entity in the meantime?

DETELE {{webApiUrl}}contacts(b8d3f910-1896-eb11-b1ac-000d3a3ac80d)
If-Match: W/"1377902"
(include the other "always include" headers)

Just like update, the above succeeds if the entity’s current ETag matches the given value and errors with a 412 Precondition Failed otherwise. In other words, if a user of your API client views an entity’s details then clicks delete for the entity, that operation will only succeed if the details they viewed match the entity’s current details. If the entity has changed in the intervening time, the delete will be rejected.

Upsert Control

If-Match and If-None-Match also can be used to change PATCH’s behavior. By default, PATH upserts—it updates the specified entity if it exits and inserts a new one if it does not.

What does a request like the below do? It depends on whether the specified entity exists. If it does, it will be updated; if not, it will be created.

PATCH {{webapiurl}}contacts(b8d3f910-1896-eb11-b1ac-000d3a3ac80d)
(include the "always include" headers)

Sometimes, this is not the desired behavior. Maybe you’re trying to update a couple of fields on an existing entity. If another user deleted the record in the intervening time, you don’t want to (re-)create a new one with just the two or three fields populated. Maybe you’d rather the PATCH fail so you can resend it setting all the fields. Maybe you’d rather it fail and not be retried out of respect for that fact that someone had a reason to delete the entity. In either case, you don’t want upsert behavior.

Using If-Match and If-None-Match, you can force PATCH to only update or only insert, instead of upserting. For more details, see tip #3 (Upsert).

The “Fine Print”

A few ETag-related details to be aware of:

  • Checking for equality is the only safe comparison to perform between ETags. Do not try greater than or less then comparisons. There’s no guarantee that an ETag from a more recent revision of a resource will be greater than the ETag for an older version—just that it will be different.
  • ETag inequality does not always indicate that a change occurred. The algorithm used to generate ETags can change with Dataverse upgrades. You may run into cases where a resource’s ETag changes even when the resource itself has not. So, while a different ETag for the same resource likely means the resource has been modified, this isn’t guaranteed.
  • ETags can affect caching (such as by a web browser). To prevent unexpected caching, every request sent to the Dataverse Web API should include an If-None-Match header, which should be set to the literal string “null” if no other value is more appropriate (see tip #1 (The “Always Include” Headers).
  • An ETag applies to a single entity. Query option $expand and header Prefer: odata.include-annotations can both result in a GET request returning data from multiple entities. When either of these is present, Dataverse won’t check the If-Match or If-None-Matched header against the entity’s ETag; instead, it will always return data (never a 304 Not Modified response).
  • ETags only work for entities where optimistic concurrency is enabled. Thankfully, by default, this option is enabled for all custom entities and some built-in ones, as well. However, this may impact you if you were hoping to use ETags with system-provided entities (like those that store internal configuration).
  • In the Dataverse world, it appears ETags are resource specific, not URI specific. I believe Dataverse will return the same ETag for an unmodified resource even if it is fetched using different URLs (e.g. if you fetch the same account using URLs accounts, accounts(guid-key) and contacts(guid-key)/my_account, the same ETag will be returned each time) even though some of the relevant specification text suggests that ETags are URI specific.

Note

Don’t forget to include the “always include” headers with every request sent to the Web API.

Reference

Perform conditional operations using the Web API

Dataverse Web API Tip #3: Upsert

• ,

In database nomenclature, an upsert (update-or-insert) is an operation that results in the specified record being updated if it exists; and, if not, a new record being inserted.

Microsoft Dataverse’s Web API PATCH operation preforms upserts by default. It’s important to be aware of what this means and, when desired, how to change it, so you’re not inadvertently upset by an unexpected upsert.

What does the below request do?

PATCH {{webApiUrl}}contacts(00000000-0000-0000-0000-000000000001)
(include the "always include" headers)

{
  "firstname": "Bob",
  "lastname": "Brown"
}

If you said that it updates the contact with the primary key of 00000000-0000-0000-0000-000000000001, setting its first and last names, you are correct if a contact with that ID exists. However, if it doesn’t—thanks to Dataverse’s identity insert behavior—the request will result in a new record being created with the given primary key, first and last name values. What you expected to be an update turned into an insert.

Is upserting a good behavior? It depends.

Continue reading →