it’s either shippable (pass) or not (fail)
Well, it seems to me that this statement is quite “theoretical”.
I am tempted to answer with a question: What is the point of “allow_failures” then?
I’m not found of the fact that I can have unnoticed failures, even allowed. Your point is that you want it to be a no-brainer “pass” or “fail”, but if there are hidden allowed failures, in my sense it’s hidden information and there is no point in running those test since they don’t count.
“Passed with warning” would still be a “passed” state, just like before, in automation context. In review context, which is intrinsically manual, it could prove very useful nonetheless. See the second use case below.
Use cases:
- On some projects, we are testing compilation against a wide variety of compilers. Some are not of prior concern. But if the test exists, It is to remind the team that this still not supported. Plus, this is only run in the “extensive testing case” not for every PR, typically.
- On the linked example, the use case is fairly different. We want to analyze the branch history at PR time, and make sure that it doesn’t introduce problematic stuff, like a binary file, or a large text file. We want to make sure of that for every commit, since the final status of the code is not enough to describe the history. This is automated in a Travis job, but sometimes a large file is legitimate. So we want to point the reviewer to the “grey zone” so that he can decide if it is a valid situation or not.