Hi Travis community!
My name is Noah and I am currently writing my bachelor’s thesis in business informatics.
My thesis is about more efficient failure cause identification by comparing two build logs.
One of the questions I want to answer in my thesis is if a specialized differencing tool supports a developer reading a build log and also if it possibly helps to find the failure cause faster.
To validate these claims, I built a tool that integrates itself as a google chrome extension into the web interface of Travis-CI.org.
The idea behind the tool is the following: When a build fails, one might derive the cause of the failure by comparing the currently failing build log with a previously successful one. The problem is, that build logs contain a lot of irrelevant data (e.g. download speeds, timestamps) that would make a traditional comparison of build logs useless. The tool I built filters for such disturbances and displays a difference between the two logs and further provides you with some useful features, to faster find the failure cause.
Now to validate the usefulness (if any) of my tool, I am relying on the experienced travis community to give me feedback.
I would like to know if you think such a tool could support you in your daily workflow or if you think this is not useful at all.
As the scope of a bachelor thesis is not big enough to support every build tool and programming language in existence, I opted for Java and Maven using Travis CI as CI solution.
Please feel free to try it out, even if you do not have a project on Travis CI available. There is a demo available on https://web.blogdiff.net/instructions#demo
If you could be a real saint, you would also participate in my very short (takes around 3min) survey (https://web.blogdiff.net/survey?source=8), but you can also just participate by replying to this thread.
I’ve seen diff filtering here and there (e.g. winmerge supports it). The problem with it is that
the specific filters needed are very situational, and,
possibly, the filters don’t make that much of a difference to justify the effort to set them up an maintain
Here’s where the devil in the details lies AFAICS: the problem is unsolvable in the general case. To really “understand” logs – to guarantee it works correctly for any possible input, – a filter needs an elaborate parser that can “read” the output of, effectively, every program in existence (and tell the output of one program from another when they are arbitrarily combined). Otherwise, (e.g. with regexes), one is stuck in an infinite break-and-fix loop.
That’s why IMO none of these solutions has gained much traction so far. But given the scale of Travis CI, easy accessibility to the solution and if you lower the setup and maintenance burden e.g. as per below and set out to primarily cover common cases (e.g. Travis-specific log artifacts – they also change fairly rarely), you may be able to make enough of a difference.
E.g.:
Make filters fully configurable and exportable.
Cover all reasonable use cases (cases like interdependent lines or filters that need to use information extracted from somewhere come to mind. Since the general problem seems unsolvable, you’ll probably have to set clear limits of what the tool is and isn’t supposed to be able to do; in any case, heed the YAGNI principle and only add engine features as new reasonable cases arise. More elaborate engine also means more elaborate, harder to write filters, so you really need to maintain the golden mean here).
Make it easy to share, collaboratively improve and combine stock sets of filters.
If any hope or solving the general problem exists, it’s here. We can only possibly cover all programs in existence if we recruit the help of all the program creators in existence.
(Last but not the least) release the solution under a free software license so that other interested parties can build upon your work without your involvement
A browser extension looks great for maximum accessibility to the solution. I’m a bit worried about a vendor lock-in though that could make it too hard to use it elsewhere. E.g. can this extension be used/relatively easily ported to other browsers?
Great idea for your thesis. I am convinced there is a need for this problem to be solved.
Years ago I tried to solve the same problem by creating a parser for travis logs https://github.com/jayvdb/travis_log_parser_wip , but didnt get very far and the output kept changing (for the better, of course). Even a loose parser wasnt sane. Doing a diff between “known good” and “new fail” logs should be more sane.
It should be easy to be travis language independent – and then I can use it
Specialised support for some languages and build tools could let the tool give better failure information, but a raw diff is still useful.
One problem to solve is mapping between a branch matrix which is different from the “last good” matrix, such as when a new job is added, or job configs are changed. Those are the times when breakages are common, even intentional, and it would be annoying if those are the times that the tool says it cant help me.
