Very interesting read and a rather "obvious" one. I can't believe I didn't see this before. Obviously... A perceptron layer is a bunch of dot products followed by comparison. Every graphics programmer knows this is a check of which side of a plane you're on.

Of course, the relu unit is also a passing on information when the result is on one side of the plane, making this a spline.

As others have said... Can we learn the separating planes without the backward gradient propagation? I don't know but seeing it in this new way may help.