Ran into this xUnit test failure the other day:

Assert.Equal() Failure
Expected: Boolean[] [True, False, False]
Actual: SelectListIterator<Process, Boolean> [True, False, False]

Why the failure?

The values displayed for expected and actual match. The only difference visible in the error message are the collection types…but that can’t be the source of failure because xUnit’s Assert.Equal<T>(IEnumerable<T> expected, IEnumerable<T> actual) doesn’t compare the collection types of its arguments, just elements and their positions.

So why the failure?

Turns out, there can be a difference between the list of values displayed in the failure message and the values that were present in each collection when they were compared.

When Assert.Equal() executes, it first determines whether the assertion passes. This involves enumerating the two collections. If that check fails, other code is invoked which renders the failure message. Producing this message involves (again) enumerating the collections. If either collection or their values changed between the failure and the failure message’s rendering, the message produced will be inaccurate. It will show values as they existed when it was rendered—which isn’t guaranteed to be the same as they were when the pass/fail check was performed.

What happened above is that actual’s values weren’t [True, False, False] when they were compared to expected, so the test failed. However, in the faction of a moment after that check but before error message assembly, actual’s values changed to [True, False, False]. When the error message was produced, it reflected these new values—which happened to match the expected values. The test failed but the error message showed actual and expected matching because they matched by the time the failure message was rendered.

The solution? Make sure that the assert’s inputs are stable enumerables (enumerables whose values won’t change). This way, Assert.Equal()’s test and error message rendering logic will operate on the same set of values. If there’s a failure, the message output will show the same values as were used for the test.

Stabilizing an enumerable is as simple as appending .ToList() to it.

Instead of:

var actual = processes.Select(p => p.HasExited;
Assert.Equal(new[] { true, false, false }, actual);

Do something like:

var actual = processes.Select(p => p.HasExited).ToList();
Assert.Equal(new[] { true, false, false }, actual);

Moral of the Lesson

Asserting on an enumerable that can change opens the possibility for race conditions to cause unexpected test behavior. Only assert on stable enumerables!