REPL and HTTP Mapping: ideas to contribute to PhantomJS

  • node • http • phantomjs • stdin • mapping • idea • repl • contribution
  • 640 words

Recently, I’m donating my spare time to the amazing Stanford Online AI Class, and writing down notes takes time.

But I’m still working, in the spare time of the spare time, on PhantomJS. Yes, very slowly, but I’m not stall.

Recently two ideas have come to mind, and it’s stuff I’d love to see supported by PhantomJS.

REPL

From the Wikipedia article:

A read–eval–print loop (REPL), also known as an interactive toplevel, is a simple, interactive computer programming environment. [...] In a REPL, the user may enter expressions, which are then evaluated, and the results displayed.

What I really like of the REPL, is the idea of launching it and start typing code into it straightaway. No files to create, save and launch. It’s very good when you want to quickly toy with a concept/idea, and it’s so small it doesn’t really deserves a dedicated script file for that.

I haven’t finished it yet, but I published my current status on this branch. Check it out!

Also, the REPL is truly enabling an even more simple concept: controlling PhantomJS by just piping commands to it through the STD_IN. A key aspect to enable easier integration of PhantomJS in larger stacks.

In facts, I discovered that sgentle has started working on wrapping PhantomJS into a Node module, and released phantomjs-node. What struck me is how he had to work out a communication channel between Node and the instances of PhantomJS: he opted for Socket.io.

Socket.io is definitely a smart solution, but phantom is launched as a child process: there must be a better way to let Node and PhantomJS speak, instead of having to pass through the network stack. Right?

And that’s where my work on the REPL (and in general into controlling PhantomJS via STD_IN) comes into the picture.

HTTP Mapping

In our effort to build proper Javascript Testing into Betfair’s new platform, a problem arises: how do we mock effectively and easily the service data responses that our code consumes?

Javascript, as many of you know, is not easy to test: it normally requires you to run your full stack to be able to test it fairly. But we are not giving up, and want to find a solution to be able to test our code, without having to run an entire infrastructure of services, particularly while code is actually developed. The developer has to be able to work into this loop:

  1. type
  2. save
  3. run test

So, here is the idea: what if we could intercept all those HTTP/XHR calls that the Javascript in the page does, and Map them to local files (or even different remotes)?

PhantomJS source code is already structured to abstract the network access of the WebPage through a NetworkAccessManager. At the moment we are using it for monitoring and sniffing.

I want to add a functionality to the webpage module so that given a file httpmap.json that looks like:

[
     {
          "method" : ["GET", "POST"],
          "source" : "http://*.betfair.com/bla/*/test",
          "destination" : "local_file.json"
     },
     {
          "method" : "GET",
          "source" : "http://*.betfair.com/bla/*/api",
          "destination" : [
               "1_local_file.json",
               "2_local_file.json",
               "http://testapi.example.com/bla/api/test"
          ]
     }
]

a tester that uses phantom could do something like:

var fs = require('fs'),
    page = require('webpage').create(),
    HTTP_MAP = "httpmap.json";

page.setHttpMap(fs.read(HTTP_MAP));

// ... do stuff with page, while the HTTP requests are mapped

As you can probably guess, the format of the map offers some extra nifty features:

  • the method field can be single (a string) or multiple (an array of string)
  • the destination field can be a single url (a string), or multiple destinations (an array of string) that phantom will cycle through at every matching request
  • the source field accepts regexp, and I’m thinking to add variables

Help

This are my ideas so far. It would be of GREAT help if you, PhantomJS lover, could spend sometime to review those and suggest accordingly.

Happy Testing!