Dataverse Web API Tip #9: Deltas (Tracking Changes)

Let’s say you want to find out about changes made to a table in Dataverse (Microsoft Dynamics, Microsoft Power Apps, etc.), perhaps because you’d like to synchronize those changes to an external system.

A primitive way of pulling this off is to query the table of interest, selecting only those records whose modifiedon value is greater than or equal to the last time you checked for changes:

GET {{webApiUrl}}contacts?$filter=modifiedon ge 2021-04-28T15:19:30Z
(include the "always include" headers)

This approach has several potential downsides, including:

• Deletes aren’t included—If a record was removed from the specified table, the results to a “GET filtered on modifiedon” query won’t return anything indicating the deletion.
• Missed modifications—Normally, modifiedon is automatically set to the moment the record was last changed in Dataverse; however, this behavior can be overridden. Since modifiedon isn’t an absolute indicator of whether or when changes occurred, filtering by it isn’t guaranteed to find all changes actually made in Dataverse after a point in time. (Think of a data import or an inbound data sync scenario: inserts/updates might be backdated so modifiedon matches when the record was last changed in the external system, which could be hours or days prior to when the data is brought into Dataverse.)

Dataverse’s Web API provides a better option for finding changes—an option built specifically for this purpose. When fetching entities from a table, if the table has change tracking enabled, you can ask Dataverse to provide you with a delta link. Later, you can use that link to fetch the delta of changes that occurred since your initial GET.

To ask for a delta link, add a preference request for odata.track-changes to your GET by setting the Prefer header:

GET {{webApiUrl}}contacts
Prefer: odata.track-changes
(include the "always include" headers)

The last page of the response will include an @odata.deltaLink.

{
  "@odata.context": "{{webApiUrl}}$metadata#contacts",
  "@odata.deltaLink": "{{webApiUrl}}contacts?$deltatoken=1408477%2104%2f27%2f2021%2017%3a00%3a53",
  "value": [
	{
	  "@odata.etag": "W/\"676509\"",
	  "contactid": "05587ab2-0596-eb11-b1ac-000d3a3ac8fa",
	   …
	},
    …
  ]
}

Later, when you want to fetch the delta of changes that occurred since your initial fetch, simply send a GET request to the saved delta link. A list of modified entities will be returned.

{
  "@odata.context": "{{webApiUrl}}$metadata#contacts/$delta",
  "@odata.deltaLink": "{{webApiUrl}}contacts?$deltatoken=1408484%2104%2f27%2f2021%2017%3a06%3a26",
  "value": [
    {
      "@odata.etag": "W/\"1408472\"",
      "contactid": "2eb46334-71a7-eb11-b1ac-000d3a32b6cd",
      "firstname": "Joe",
       …
    },
    {
      "@odata.context": "{{webApiUrl}}$metadata#contacts/$deletedEntity",
      "id": "5028a3de-70a7-eb11-b1ac-000d3a32b6cd",
      "reason": "deleted"
    }
  ]
}

Deletes are represented in the collection by simple JSON objects having an @odata.context indicating deleted entity and a reason of “deleted” (see last entry in the above results example).

The last page of the delta response itself includes a new delta link. You can use this link to fetch subsequent changes (that is, changes occurring after this delta). If you want to repeatedly check for changes over time, the cycle is: perform an initial GET with the odata.track-changes preference, then save that delta link and use it the first time you check for changes, then use that response’s delta link the next time you want changes, then use that response’s delta link for the next check, and so forth.)

If you’d simply like the count of change entries in a delta, append a /$count right before the query string (only works if $select isn’t in use):

GET {{webApiUrl}}contacts/$count?$deltatoken=1408484%2104%2f27%2f2021%2017%3a06%3a26
(include the "always include" headers)
	
Response: 
2

Note that the returned count maxes out at 5,000. A response of 5,000 should be interpreted as “5,000 or more changes.”

A few things to be aware of:

• Change tracking must be enabled on the table (as already mentioned).
• The delta link only returns changes for the specified table, not related tables.
  • Example: A delta link for contacts will return new and updated contacts (including contacts who have had relationships changed), as well as deleted contacts. However, the delta will not indicate whether changes to related entities occurred—like whether the name on an account related to a contact was modified.
• A delta fetch may include deletes for entities that were created then deleted after the delta link was generated.
  • Example: You fetch a delta link for a table. After you obtain the link, someone creates then deletes an entity out of the table. When you use the link, you may see a delete entry for that entity even though you weren’t previously aware of its existence (as it didn’t exist when you fetched your delta link).
• Technically, it’s possible for delta fetches to return unchanged data. The delta link promises to at minimum return changes; it doesn’t promise that everything it returns was changed.
  • Example: You have a delta link from a GET request that included a $select column list. The link may return entities where the columns in the $select list haven’t changed but where other columns were modified (i.e. properties outside of the $select list were updated).
• If you don’t use a delta link for an extended period of time (one month?), the link may not work. Unfortunately, old delta link behavior doesn’t seem to be clearly documented (at least, from the reading I’ve done). It appears that using an expired delta link possibly may return a 400 error complaining that the “Version stamp associated with the client has expired. Please perform a full sync” (code: 0x80044352).
• Technically, the OData spec says that if you want a delta fetch’s results to include a new delta link, the fetch should include the Prefer: odata.track-changes header. However, Dataverse’s Web API doesn’t (currently) require this: it always returns a new delta link anytime a delta fetch is performed. For future compatibility purposes, you might consider including this preference when you perform a delta fetch if you want a new delta link.

Note

Don’t forget to include the “always include” headers with every request sent to the Web API.

Reference

Use change tracking to synchronize data with external systems