Fun with brute-force testcase reduction


Good idea, guys!

I never expected any success when I posted my little Remynifier project to Hacker News. I made front page, on my very first submission ever.

There was a lot of good feedback there. I really liked the suggestion to report some of the bugs that I found.

I don't really know CSS, nor can I use JavaScript, so that would make debugging Node applications running over CSS a bit difficult. Still, I am happy to take on a challenge.

Except, I'm quite busy and so I decided I'm going to do it my way. I wanted to have a laugh first, get something done second, all this using the least amount of my time possible. You know, the typical Remy approach.

Here's what I did.

I put on my raincoat and deerstalker hat

First, I had to find which chain of minifiers was going to hang on me.

Easy enough. I took the timeout limiter off, I tweaked the Remynifier to print the tasks as they are being taken out of the work queue. I saw that it may take a little bit of time for the Remynification to hit upon the hang that I saw in the past.

I did the most obvious thing to speed up the process: I took a break for lunch.

When I got back I saw that the problem is caused by running the chain of animate.css>cssnano>csso.

At this point I could just call it all a day, but I wanted to be a bit more helpful than just throwing a huge file in the general direction of the csso maintainers. I'm not trying to win the "least helpful bug report that at the same time contains absolutely all of the information that the maintainers need to fix the problem" contest. Not this time, anyway.

My lack of skills eliminated my ability to do something intelligent. Thankfully, it isn't like a little hurdle such as "I have no idea what I'm doing" has ever stopped me before. I did other things for the rest of the day and then, in the middle of the night, I got an idea.

Let's get pretty!

I needed the CSS to be very sparse.

Accordingly, the first step was to make the file pretty. I threw it into some random prettyfier. Out came a file of about 2.8K lines.

Here's a sample:

animated {
    -webkit-animation-duration: 1s;
    animation-duration: 1s;
    -webkit-animation-fill-mode: both;
    animation-fill-mode: both
}
.animated.infinite {
    -webkit-animation-iteration-count: infinite;
    animation-iteration-count: infinite
}
.animated.hinge {
    -webkit-animation-duration: 2s;
    animation-duration: 2s
}
... Much more here ...

I verified that the expanded version still hangs in processing.

Good!

Surgical strike

The process I had imagined was simple. We know the input is bugged. We will try to automatically fix it. We can use the knowledge of the fix to guide what gets sent to the maintainers.

How to automatically fix buggy inputs, you may ask?

Easy! We can take inspiration from the precision work of doctors in my home town. Their approach to diagnosing seemed to be to remove the pieces of the patient until the patient stopped complaining. That part that was last removed was declared to have been the cause of the complaints. (They found a surprising amount of heart disease and mental illness.)

Likewise, we will rip out random lines from the file until it is small enough. If csso times out on this input, then we have found the new smallest case that demonstrates the problem. If not then we rip out a different random line. Repeat. Forever.

I set a timeout for five seconds. With a bit under three thousand lines and at five seconds per timeout, this would take a while, not even counting the fact that a line can be unsuccessfully chosen for removal multiple times.

I did the most obvious thing to speed up the process: I went to sleep.

Good morning!

When I opened my eyes, I saw that the CSS file was at 14 lines. Progress!

.animated {
}
.animated.infinite {
}
.animated.hinge {
}
.animated.flipOutY {
}
    40% {
    }
    60% {
    }
.fadeIn {
    -webkit-animation-name: u;
}

I wasn't expecting this much progress, to be honest. I was expecting many irrelevant properties and values cut, but not whole chunks of the file. This made me think that this was a parser error.

I knew enough of CSS to be able to cut this down by hand.

.fadeIn {-webkit-animation-name: u;}

Does the selector matter, or is the problem caused by the property being set? Turns out it was neither. The value that is being set is the cause of the problem.

p{B:u}

This is what I submitted as an issue on Github. A new release was out within an hour. I was highly impressed by the quick response, though I wonder if enough QA was done to make sure that the fix does not impact the rest of the software.

I worry too much. It was just a one line fix, so there probably isn't a problem with it and the regression tests would have caught it anyway.

The right way to do this

I coded up the line remover at about 02:00. I needed a little bit of satisfaction and amusement to put me to sleep after a tough day. I had a laugh at how completely stupid my Automated CSS Repairman was and I went to bed, not expecting any meaningful results. I woke up to meaningful results.

There were many reasons why this worked. Simply put: I was really lucky.

First of all, it was an infinite loop that I was trying to track down. If it were an error of some other kind then removing lines would possibly replace one error with another, making it impossible to trivially see if I'm still tracking the original issue or one caused by line removal. Hangs tend to be easy to identify. Worst case scenario would have me report a different infinite loop than the one I started with, but those are always good to fix. The process then could be iterated.

Second of all, the bug was caused by a fairly isolated chunk of CSS. If the error had been spread all around the CSS, I would not have been able to reduce it so nicely.

Third of all, the error was really close to the start of the file. Any lines that were past the hang could have been removed without being examined by csso, even if they would have otherwise caused a problem by being absent. The hang took precedence. This really sped up the removal of lines.

Fourth of all, I had to go to sleep anyway, so the runtime wasn't an issue. I could afford the computer to spin for eight hours. My scaffold was strong and I had enough rope for the hangings too. If resources had been an issue, I would have been more fancy about this.

There is a much better way of doing this.

I could have attempted to remove each line once, systematically going from the start of the file to the end. Then I could have gone for adjacent pairs, adjacent triples and so on.

Tired Remy did not think of any of that, even though it was completely clear to Rested Remy. I took enough courses on input minimization to be able to come up with something better than a completely random solution. In the end I was just too exhausted to think of it.

Thing is, the idea, horrible as it was, worked.

Closing thoughts

It took me about fifteen minutes, total, to submit the bug report. I wasted a ton of computer cycles, but my own time was saved. I like that.

I'm starting to believe in the concept of Minimum Viable Program. I'm not sure if this is a good thing.

Medicine in small towns kinda sucks.

If you want me to keep writing stuff, give me a few coins to keep me motivated!