Controlling Firefox from Perl with MozRepl

June 24, 2012

On Friday I wrote my first program using AnyEvent and Coro, and it was so nifty that I decided to revive my moribund blog to describe it. This little program solved a whole raft of problems.

The problem involved determining certain properties of each of a list of URLs and updating a database. The rub is that some properties, such as redirects, are most reliably observed in the browser, and some, such whether the URL is hosted at $WORK, best determined using $WORK’s Perl modules. Now how can we put these two together to get all the info we need?

The approach I hit on involves using Perl to drive an instance of Firefox configured with some helpful extensions. MozRepl is a cool extension that lets you telnet into Firefox and program it from the inside, with access to the entire browser and the Mozilla API. NetExport is an extension to the extension Firebug which generates HAR (HTTP archive) files capturing all the data necessary for the analysis of front-end performance. HAR files serve as input to the command-line versions of PageSpeed and YSlow.

NetExport exports its results either to file or via HTTP POST. So in my program I start an embedded HTTP server, fire up Firefox, telnet into it, load a URL, and let the httpd capture and process the JSON posted by NetExport. Think of the setup as making Firefox puke into a bucket set out for that purpose. A bit imagé but you get the idea.

How do AnyEvent and Coro enter the picture? I use AnyEvent condition variables to coordinate the work between page loading with MozRepl and output handing with the web server, and a Coro thread to tell the web server to kindly stand over there while I continue with my main line of work. The hardest part is reorienting one’s brain towards the asynchronous way of thinking, not so easy after a lifetime of vanilla scripting.

So let’s see some code. The main script is delightfully short and sweet; the annotated version follows. The name of my employer has been changed to protect the innocent. And of course I use strict and warnings.


use AnyEvent;
use Coro;
use WORK::Config;
use WORK::AnyEvent::HTTPD; 
use WORK::AnyEvent::HTTPD::Handler::NetExport;  
use WORK::AnyEvent::MozRepl;
use Log::Log4perl qw(:easy);

my $cfg = WORK::Config::get_config();

# AnyEvent condition variable 

my $cv  = \$WORK::AnyEvent::HTTPD::Handler::NetExport::cv;

my @urls = qw(http://www.google.com http://www.yahoo.com);

# Tell NetExport where to post its results.
my $beacon = "http://localhost:9090/netexport";

# Handlers to process POSTed results.

my $h = WORK::AnyEvent::HTTPD::Handler::NetExport->new;

# Fire up a server and ask it to step out of the way.
async { 
    start_httpd($h);
}

# Fire up FF with good ol' system().

start_firefox();

# Connect to MozRepl with AnyEvent::Socket

mozrepl_connect();

# Set the POST URL and turn on auto export.

set_netexport_prefs($beacon);

while (@urls) {

    $$cv = AnyEvent->condvar; 

    my $url = shift @urls;

    load_page( $url ); #  Using MozRepl

    $$cv->recv;   # The POST handler will send().
    
    clear_cache(); 
}

kill_firefox();  # From within MozRepl.

Neat huh? There’s lots of blanks to fill in, but as this post is already getting a bit long, I will do that in posts to follow.

Advertisements

Yes!

October 17, 2009

It is my pleasure to announce that after much fooling around with XS, I am now reading temperatures from the device with Perl.

ok 1 - use Vernier::GoTemp;
ok 2 - The object isa Vernier::GoTemp
# 27.4995159422979
# 27.4995159422979

That is the ambient temperature in Celsius.

Update: My ten-year-old son and I had a grand time measuring temperatures under our armpits, behind our knees, in front of the fan, and in glasses of ice, lukewarm and warm water.


Devel::Peek

October 17, 2009

Today I belatedly discovered the usefulness of Devel::Peek.

While hacking on the newly-rebaptised Vernier::GoIO I found that get_device_info was handing me back garbage. Using Devel::Peek helped me pinpoint the problem more quickly.

get_device_info should return an array consisting of device name, vendor ID and product ID; the former is a string in hex and the latter integers required by HID USB.

Devel::Peek to the rescue. At the top of Vernier/GoIO.pm:

use Devel::Peek qw(Dump);

and in the offending routine, a strategically-placed call to Dump:

sub get_device_info {

    my $handle = shift;

    my @info = Sensor_GetOpenDeviceName($handle);

    Dump(\@info);

    die "No device info found for $handle!" unless @info;

    return (
        'name'       =>  $info[0],
        'vendor_id'  =>  $info[1],
        'product_id' =>  $info[2],
    );
}

Running a test nets me this output:

SV = RV(0x82603c) at 0x86590c
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x8633bc
  SV = PVAV(0x864684) at 0x8633bc
    REFCNT = 2
    FLAGS = (PADBUSY,PADMY)
    IV = 0
    NV = 0
    ARRAY = 0x30b9e0
    FILL = 2
    MAX = 3
    ARYLEN = 0x0
    FLAGS = (REAL)
    Elt No. 0
    SV = PV(0x849ad4) at 0x866074
      REFCNT = 1
      FLAGS = (POK,pPOK)
      PV = 0x304eb0 "0x1d100000"
      CUR = 10
      LEN = 12
    Elt No. 1
    SV = IV(0x80d3ac) at 0x865954
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 8388740
    Elt No. 2
    SV = IV(0x80d380) at 0x8660c8
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 8388740

An array ref with three elements. Looks good. But those last two values? Not what I had in mind. I should get 2295 and 2, respectively.

perlguts gave me a clue:

Despite their suggestions in earlier versions of this document the macros (X)PUSH[iunp] are not suited to XSUBs which return multiple results. For that, either stick to the (X)PUSHs macros shown above, or use the new m(X)PUSH[iunp] macros instead; see “Putting a C value on Perl stack”.

I’d pushed three scalars on the stack, the first using the macro XPUSHs and the next two with XPUSHi:

            XPUSHs(sv_2mortal(newSVpv(name, strlen(name))));
            XPUSHi(sv_2mortal(newSViv(vendorId)));
            XPUSHi(sv_2mortal(newSViv(productId)));

I changed the XPUSHi to XPUSHs as advised by perlguts, re(generated|compiled|tested) and lo! correct return values.

SV = RV(0x82603c) at 0x86590c
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x8633bc
  SV = PVAV(0x864684) at 0x8633bc
    REFCNT = 2
    FLAGS = (PADBUSY,PADMY)
    IV = 0
    NV = 0
    ARRAY = 0x30b9e0
    FILL = 2
    MAX = 3
    ARYLEN = 0x0
    FLAGS = (REAL)
    Elt No. 0
    SV = PV(0x849ad4) at 0x866074
      REFCNT = 1
      FLAGS = (POK,pPOK)
      PV = 0x304eb0 "0x1d100000"
      CUR = 10
      LEN = 12
    Elt No. 1
    SV = IV(0x80d334) at 0x865954
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 2295
    Elt No. 2
    SV = IV(0x80d3ac) at 0x8660c8
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 2

perlguts has this to say about why those two integer values were identical:

The following code will not do what you think:

 XPUSHi(10);
 XPUSHi(20);

This translates as “set TARG to 10, push a pointer to TARG onto the stack; set TARG to 20, push a pointer to TARG onto the stack”. At the end of the operation, the stack does not contain the values 10 and 20, but actually contains two pointers to TARG, which we have set to 20.

Okay then. Point taken, and mental note made to study
“Putting a C Value on the Perl Stack” more closely.


Typemapping

October 11, 2009

Today’s XS problem was solved with a tweak of the typemap.

I wrote a constructor for the Go! Temp device class Vernier::GoTemp that, amongst other things, ‘opens’ the device; that is, establishes communications with and initializes it.

The C prototype is

GOIO_DLL_INTERFACE_DECL GOIO_SENSOR_HANDLE GoIO_Sensor_Open(
        const char *pDeviceName,        
        gtype_int32 vendorId,   
        gtype_int32 productId,  
        gtype_int32   strictDDSValidationFlag);

and the XSUB is


GOIO_SENSOR_HANDLE
GoIO_Sensor_Open(pDeviceName, vendorId, productId, strictDDSValidationFlag)
        const char *    pDeviceName
        gtype_int32     vendorId
        gtype_int32     productId
        gtype_int32     strictDDSValidationFlag

The make test error I got was

pDeviceName is not of type const charPtr at /Users/zrusilla/proj/goio/Vernier/blib/lib/Vernier/GoIO.pm line 150.

The device name is a hexadecimal number 0x1d100000, so I thought the integer part of the scalar was being passed long to the XSUB, which howled in protest. How to fix? With a tweak of the typemap.

The previous entry for a const char *was
const char * T_PTROBJ

The typemap documentation tells me:

T_PTROBJ
Similar to T_PTRREF except that the reference is blessed into a class. This allows the pointer to be used as an object. Most commonly used to deal with C structs. The typemap checks that the perl object passed into the XS routine is of the correct class (or part of a subclass).

The pointer is blessed into a class that is derived from the name of type of the pointer but with all ‘*’ in the name replaced with ‘Ptr’.

Ah, so that’s the problem. When the C header says char *, it means char *. So I changed the typemap entry to a T_PV, which is a plain old char *:

const char * T_PV

rebuilt and tested. And all tests pass.


Another day, another XSUB

October 7, 2009

Back to the GoIO project.

I’ve got a C subroutine

gtype_int32 GoIO_GetNthAvailableDeviceName(
        char *pBuf,  //[out] ptr to buffer to store device name string.
        gtype_int32 bufSize,  //[in] number of bytes in pBuf
        gtype_int32 vendorId, //[in] USB vendor id
        gtype_int32 productId, //[in] USB product id
        gtype_int32 N);   //[in] index into list of known devices
                                 // 0 => first device in list.

that I wish to call from Perl as:

my $name = GetNthAvailableDeviceName(vendorID, productID);

dispensing with the need to pass in a string buffer. The XSUB to do this is:

NO_OUTPUT gtype_int32
GoIO_GetNthAvailableDeviceName(gtype_int32 vendorId, gtype_int32 productId)
    PREINIT:
        char s[GOIO_MAX_SIZE_DEVICE_NAME];
        gtype_int32 found;
    PPCODE:
        found = GoIO_GetNthAvailableDeviceName(s, GOIO_MAX_SIZE_DEVICE_NAME, vendorId, productId, 0);
        printf("%d\n", found);
        printf("%s\n", s);
        if (found == 0) {
            EXTEND(SP, 1);
            PUSHs(sv_2mortal(newSVpv(s, GOIO_MAX_SIZE_DEVICE_NAME)));
        }

The test passes, returning the device name in glorious hexadecimal:
t/05_available_device….# 0x1d100000

Exciting stuff, huh?

I think I’ll go watch some grass grow now.


Pulp Perl

October 6, 2009

 

Enough Rope to Shoot Yourself in the Foot

Enough Rope to Shoot Yourself in the Foot

 

 

No XS revelations tonight. Just a new ‘toon.


Pulp Perl

October 5, 2009

Far out, it’s a new Pulp Perl cartoon!

Hippie hack-in

Hippie hack-in