The Clairvoyant project is one of my more ambitious personal projects, with one “simple” goal in mind: Your tests should be able to generate your application code
This post will outline the beginnings of the madness that led to Clairvoyant as well as some of the details of how things are planned to be implemented.
Code as Data
The LISPers among you will notice a very common theme throughout this post. That theme is quite simply that I’m taking a ruby file and treating it as data for an entirely different parser.
The DSL is already there, the data set, the question becomes what can we divine from what we have with reasonable certainty?
Along with inspirations from LISP, we’re drawing pretty from Logical languages such as Prolog. A logic program is a statement of facts used to derive an answer to a question:
1 2 3 4 5 6 7 8 9 10
Now I don’t pretend to be an expert in Prolog, or even really particularly any good at it. Given that, it still reminds me of something:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
So if RSPEC looks like it’s already making assertions about the nature of our program, what happens if we treat it like a logic language?
Repurposing a DSL
The thing about Ruby is it’s incredibly flexible. The DSL from RSPEC can just as easily be hijacked and run in another compiler with this one simple trick (and I swear I won’t clickbait):
1 2 3 4 5 6
Now what have we done? We’ve evaluated the entirety of the loaded file in the context of our class. What happens if we redefine describe inside of there?
1 2 3 4 5 6 7 8 9
We can capture the entirety of the
describe blocks, or for that matter anything else we want. As long as the spec file isn’t using
::RSpec.describe we can hijack whatever we want. If it does, it just makes it mildly more annoying to reason about.
This is effectively the current state of Clairvoyant. You can take a very basic spec file and generate a skeleton of it such that:
1 2 3 4 5 6 7
1 2 3 4 5 6 7 8
Granted that’s not all that impressive at this point, but beyond this stage we’re going to find some very interesting problems. This brings me to my next section.
Generating a skeleton is easy enough, and still very useful in its own right. Actually writing code from expectations and matchers that could be near infinite in number and complexity? That becomes a whole different story very quickly. These are theoretical musings of the future nature of Clairvoyant as I see it.
The nature of the ‘it’ block
Past the description, we’re defining what the logic of the program is. From here we can infer a good number of relevant details:
1 2 3
The description string of the method can be used for documentation and a nifty method description in some cases.
The actual expectation call tells us the name of the method, and potentially anything that we can call after it. In the above example we know that
last can be called on the result of our method and the
arity of the method can be
3. We can also guess that the method is some form of Enumerable, dropping possible options for output substantially. Given that we only have an integer here, we can make a reasonable statement that the return value of method is
We have facts to work with here, and the more
it methods we have inside of a
describe, the more we can divine from given facts of the method. Say another test called
keys on our methods return, now we can reasonably guess it’s a
Hash or close derivative.
That’s well and good, but a lot of ruby methods tend to be very conditional in nature. They could return different things dependent on a
context. Luckily we have just such a method we can hijack!
Contextual contexts abound
Say we find our method doing strange things dependent on what the
context is. We can possibly even derive a conditional from a well laid out
1 2 3
We have a potentially bindable name in
value, and a proper context test to throw against with
is even. Of course at this point it’s going to be a lot more difficult to glean this information and will be heavily dependent on the robustness of a tokenization library and parser.
This becomes substantially more difficult to reason about, because now we’re trying to tell people how to write their tests instead of divining information from what’s already there. That may be fine for new code but can be incredibly tedious to make behave properly.
Matchers provide an even more interesting challenge, especially factoring in user defined options. We can make some reasonable assertions based on
raise error to give some error handling for dynamic methods, but that again becomes dependent on
context blocks being clear enough to grok.
Given that we have all of that figured out, the next fun part is proving whether or not what we did even works right. At this point we can run our generated code through the RSPEC again as the core team intended to see if we made it pass. If we did, great! That’s the easy case if we’ve already gotten this far. If we haven’t on the other hand it opens up a whole different can of worms.
So maybe it didn’t pass. Hey! It gave us data back to use to further refine and polish our solutions. We can use that to (hopefully) get them to pass on the next round! At this point the failed tests would be ported back and we could do a few things at this point:
Fail the method and leave a comment for the user or Attempt to repair the method to make the test pass
The first would be far more practical if we’ve gotten this far, but hey, we’re in theory land. Let’s push our luck a bit more here.
At this stage we would be throwing code back and forth until something works, a very brute force solution to hoping we hit the sweet spot. In something that could only be compared to the quandry of monkies writing Shakespear, we might squeeze just a bit more code out of there.
Though honestly, halting problem is just a euphemism for being dull, let’s have more fun!
So we can get a hold of a lot of your application code as well right? Let’s not limit that. Let’s grab as much ruby code as we can stuff into memory and try and find patterns between their RSPEC code and application code. Machine learning and deep analysis can be applied to more acurately divine intended code based on community behaviors (though I will explicitly prune out you maniacs who use globals like candy.)
Throw it in a Spark cluster and let the thing roar. We’re deep into AI land of making some very interesting code generation black magic happen, and probably well beyond anything that’s been attempted up to this point.
I have no qualms saying this is well beyond me, but it sounds like a blast to try anyways.
You’re out of your mind
It wouldn’t be the first time I’ve been told this, and certainly won’t be the last. This is a personal project and a great deal of fun in learning Ruby internals along the way. Maybe one day this will be a fully functional project that can magically make your wildest dreams come true, or maybe not.
Really, when it gets down to it, that’s the fun of it. The potentials here are limitless, the problem hard, and the code plentiful. That’s the best type of problem to poke at. It’d be no fun if I knew entirely what I was doing.
Check out Clairvoyant, leave me a comment, let me know what you think!