New M Feature: Structured Error Messages

, , ,

Why Structured Error Messages?

In the real world, errors are a part of life. If you access and read data from real, in-production systems, sooner or later you will almost certainly encounter errors. While you may be unable to escape their unfortunate reality, at least in the Power Query world, they’re rendered out in an easy-to-read format:

Expression.Error: Bad code 'ABC', problem 'too short'

Easy to read, that is, if you are a human, reading just one error all by itself.

But what if you’re trying to analyze a collection of error messages? Imagine a set of errors like the above, but which are for a variety of different codes and problems (e.g. bad code ‘A235’, problem ‘must contain at least 2 letters’, bad code ’15WA’, problem ‘cannot start with a number’, etc.).

Let’s say you want to summarize these errors, reporting out the count of errors per problem, per bad code. Manually reading errors one at a time no longer cuts it. Instead, you could write code that parses each error message, extracting the text between the phrase bad code ‘ and the following quote character, and between problem ‘ and the following quote character. With the code and problem statement now separately captured, you can use their values to group by or otherwise compute the desired summaries.

Parsing log messages like this this involves coding work. Not only does it take effort on your part, but it is also tricky to get right. For example, the logic described above finds the end of each string it matches by looking for the next quote character. What if a bad code or problem description includes a quote character? The logic we’ve been considering won’t match the entire value. Say, the message starts with Bad code ‘ABC’DEF’. The above logic will miss the second portion of the code (only capturing ABC, not the full ABC’DEF) because it incorrectly assumes that a bad code will never contain a quote. You could address this by writing more robust parsing code, but that’s more work—and this is only one example of the corner cases you may need to handle to accurately parse a family of log messages.

On the other hand, maybe your interest is not analyzing log message parameters, but rather removing them altogether. For data privacy or security reasons, you want sanitized log messages, where parameter values have been stripped out and replaced with generic placeholders. This way, “clean” log messages can be aggregated or retained long-term without the complications that accompany storing PII or other confidential information that may have found its way into error message parameters. While this may be the opposite of our first scenario (extracting message parameters for analysis purposes), implementing it still requires a technical means to differentiate between the base log message pattern (or template) and the parameters that have been filled into it. If you’re implementing this yourself, you’re looking at some form of log message parsing.

In either case, if only there was a way to avoid the effort and complexity associated with writing log message parsing code….

Introducing M’s Structured Error Messages

Meet M’s new structured error message capabilities!

M’s error functionality has recently been expanded to offer a new way of defining error messages, splitting message definition between a template and a list of parameter values. These components are preserved with first class representation in the error after it is raised, enabling error handling code (and, potentially by extension, external logging mechanisms and log analytics tools) to separately work with these components without the need for custom text parsing. This style of error message is known as a structured error message and is key to making structured logging possible.

How It Works

When raising an error, the error definition record now supports two new fields:

  • Message.Format—A string defining the error message’s template.
  • Message.Parameters— A list of values used to fill in the placeholders in the message’s template.

When an error is raised with Message.Format set to a non-null value, M will generate the error’s Message by performing string interpolation using Message.Format as the template. Message.Format may contain placeholders in the form of #{x} (where x is a 0-based index). These placeholders are replaced with the corresponding values, located by index, from the parameters list. The resulting string becomes the Message for the error that is propagated.

let
  RaisesError = error [Message.Format = "Bad code '#{0}', problem '#{1}'", Message.Parameters = {"ABC", "too short"}] 
in 
  RaisesError
Expression.Error: Bad code 'ABC', problem 'too short'

To the human reader, this looks identical to an old-school, non-structured error message. The difference—and it’s a significant difference—is behind the scenes.

In M, if you catch an error with a structured error message, the error’s record will contain the fully rendered error message (in field Message) as well as the original message template (in Message.Format) and the message parameters (in Message.Parameters).

let
  RaisesError = error [Message.Format = "Bad code '#{0}', problem '#{1}'", Message.Parameters = {"ABC", "too short"}], 
  CaughtError = try RaisesError
in
  CaughtError[Error]
Screenshot of an error record showing Message.Format and Message.Parameters

The net effect is a “best of both worlds” scenario: A human user encountering the error in the UI still sees the error’s Message, which in the case of structured error messages was generated from the message template by filling in its placeholders with parameter values. For programmatic processing from within M, the rendered message, message template and parameters are available separately and so can be individually accessed without the need for custom text parsing.

For example, if you catch an error in M that is defined using a structured error message, you don’t need to parse the Message to extract its parameters; instead, you can simply read them out of Message.Parameters.

let
  RaisesError = error [Message.Format = "Bad code '#{0}', problem '#{1}'", Message.Parameters = {"ABC", "too short"}], 
  CaughtError = try RaisesError,
  BadCode = CaughtError[Error][Message.Parameters]{0}
in
  BadCode // returns ABC

Hopefully, these new error fields (Message.Format and Message.Parameters) will eventually be made available in the logging and query diagnostics that Power Query outputs.

Additional Details

Now that you have the big picture of this new feature, time for some fine-point details:

  • If an error is raised with both Message and Message.Format specified, the specified Message will be ignored as the Message propagated with the error will be generated from Message.Format.
  •  When an error message is generated from Message.Format, behavior equivalent to Text.Format is used to render the parameters to text (reference). In a nutshell, simple scalar values are rendered to text, while complex values (like tables and lists) are represented in the interpolated string by their literal type name. For example, if a table is used as a parameter, it will be output simply “table” in the generated message.
  • If an error is raised with a Message.Parameters list but without specifying a Message.Format, the parameters list will not be included in the error that is propagated.
  • This new functionality is optional; raising errors with Message instead of Message.Format + Message.Parameters remains fully supported.
  • This new feature as no effect on error fields Reason and Detail, which can be used with either style of error message. The latter still remains the preferred location for in-depth details about an error (if any).

Best Practice Recommendation

When writing new M code, if you find yourself raising an error whose message consists of both static text and parameter-like values, strongly consider using a structured error message.

Leave a Reply

Your email address will not be published.