Benchmarking Magic (revisited)

Since I started development on Garden (the next Lussumo development framework), I have spent a LOT of time crawling the web for information on everything from PHP optimization, to wacky new design patterns; anything I can get my hands on that will spark some new ideas in my head. In my travels, I’ve found a few very interesting posts that I’ve gone back to and read over and over again; getting new insights with each new read (and subsequent contemplation). One of these articles is called Benchmarking Magic by Larry Garfield (If you are interested in PHP development, I highly recommend you read his blog regularly).

I’ve been entranced by the possibilities of PHP’s magic methods for some time; but all I’ve ever heard about them was that they are slow. Larry took the time to benchmark PHP’s magic methods next to the standard alternatives to see exactly how slow they are. His results, though not too surprising, were definitely enlightening. Calling PHP’s magic __call method is slow, but the real issue is that you can’t do much with a __call unless you are then able to take the requested method and arguments and do something with them. This usually involves the use of PHP’s call_user_func or call_user_func_array functions. And Larry’s results showed that while the use of __call was slow, if you combined __call with the use of call_user_func*, it resulted in an abysmally slow return.

Now, I’ve been struggling to come up with a slick way of integrating plugins with Garden, and I decided to take the plunge and use PHP’s __call magic method to help me accomplish part of the plugin framework. But after reading and re-reading Larry’s article (and this other one by Paul M Jones) I decided that I just can’t justify using either of the call_user_func* functions – it would simply be too slow!

But, what are the alternatives?
The __call method takes two arguments: $Method and $Arguments. These represent the name of the method that was called, and the arguments that were passed into that method. The real problem, and the reason why you’d need to use call_user_func_array() 99% of the time is because when you redirect that method call somewhere else, you need your arguments to be split so that they go into the resultant method in individual pieces rather than as one single array called $Arguments.

In other words, if I am redirecting a method call to another method on the request object, this would work:

function __call($Method, $Arguments) {
     call_user_func_array(array($this, 'DestinationMethod'), $Arguments);
}

But this would NOT work:

function __call($Method, $Arguments) {
     $this->DestinationMethod($Arguments);
}

Because DestinationMethod is expecting the arguments to come in as:

$this->DestinationMethod($Arguments[0], $Arguments[1], $Arguments[n]);

But in a general sense within the constructs of Garden, I might not always know what the “DestinationMethod” will be (implying I’ll end up having to reference the destination method as a variable variable, like $this->$DestinationMethod()), let alone how many arguments that method is expecting!

So, last night I was thinking about how I could possibly work around the necessity of call_user_func_array(), and a wacky idea came to me: Why not pad the $Arguments array up to a length of 10 and pass in all 10 arguments to the variable call of $DestinationMethod? Sure, $DestinationMethod might only have 3, 4, or x number of arguments, but since I have full control over how many arguments go into all of the methods in Garden, I can certainly examine it to ensure that there are never more than 10; and PHP doesn’t care if you don’t define extra arguments explicitly.

For example:

function Foo() {
     return;
}

// This does not result in any errors being thrown:
Foo(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);

But the question remains: How much faster is a variable method call than a call_user_func_array call?

Using similar methods to Larry Garfield, I created a benchmarking script to test everything from a literal function call straight down to a variable method call via PHP’s __call magic method. Just like Larry’s results, my exact numbers in the following tests weren’t particularly interesting; What’s interesting is their relative value. All tests are run in a single script (below) on the following system:

Macbook Pro on AC Power
Intel Core Duo 2 GHz
2 GB RAM
Apache 1.3.39 on Windows XP SP3
PHP 5.2.5

To quote Larry:

Because it’s a fairly beefy system, all tests are run 2,000,000 times so that we have worthwile numbers to compare. All times listed below are in seconds. Of course, any such tests will vary a bit between runs, and even between two tests in the same script. We’re looking for overall trends here, not exact numbers, but it’s important to keep in mind that micro-benchmarks are an inexact science. Also keep in mind that I don’t know the internals of the PHP engine well at all, so my analysis is based on logical extrapolation, not actual knowledge of the PHP engine itself.

The Results

Testing Method Time
Literal Function 1.941
Variable Function 2.792
call_user_func() 5.445
call_user_func_array() 6.376
Literal Method 2.912
__call 6.931
__call with call_user_func_array 17.903
__call with Variable Method 10.727

There you have it, a simple variable method call is most definitely faster than call_user_func_array(), and you can overcome the necessity of call_user_func_array by passing in x number of arguments and letting the extras “go to waste”.

So, did this solve all of my problems with the addon framework in Garden? Nope. But it lets me sleep a bit easier. I’m sure that resorting to __call on every method call would result in a slow application; but I doubt that plugin authors *need* that much access to the core. So it all comes down to deciding where it would be most useful to implement in the application.

For anyone interested, here is the script that gave me the results above:

<?php
error_reporting(E_ALL | E_STRICT);
define('ITERATIONS', 2000000);

function Foo($Bar) {
   return;
}

$Foo = 'Foo';

class TestCall {
  function Foo($Bar) { return; }
  function __call($Method, $Args) {
   if ($Method == 'VariableMethod') {
      $Call = 'Foo';
      return $this->$Call($Args[0]);
   } else if ($Method == 'Return') {
      return;
   } else {
      return call_user_func_array(array($this, 'Foo'), $Args);
   }
   return;
  }
}

$t = new TestCall();

function Row($Test, $Start, $Stop) {
   echo '<tr><td>'.$Test . '</td><td>' . ($Stop - $Start) . ' seconds</td></tr>' . PHP_EOL;
}

echo '<table border="1" cellpadding="0" cellspacing="3">';

$Start = microtime(true);
$TestName = 'Literal Function';
for ($i = 0; $i < ITERATIONS; ++$i) {
   Foo(1);
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

$Start = microtime(true);
$TestName = 'Variable Function';
for ($i = 0; $i < ITERATIONS; ++$i) {
   $Foo(1);
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

$Start = microtime(true);
$TestName = 'call_user_func()';
for ($i = 0; $i < ITERATIONS; ++$i) {
   call_user_func('Foo', 1);
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

$Start = microtime(true);
$TestName = 'call_user_func_array()';
for ($i = 0; $i < ITERATIONS; ++$i) {
   call_user_func_array('Foo', array(1));
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

$Start = microtime(true);
$TestName = 'Literal Method';
for ($i = 0; $i < ITERATIONS; ++$i) {
   $t->Foo(1);
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

$Start = microtime(true);
$TestName = '__call';
for ($i = 0; $i < ITERATIONS; ++$i) {
   $t->Return(1);
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

$Start = microtime(true);
$TestName = '__call with call_user_func_array';
for ($i = 0; $i < ITERATIONS; ++$i) {
   $t->CallUserFuncArray(1);
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

$Start = microtime(true);
$TestName = '__call with Variable Method';
for ($i = 0; $i < ITERATIONS; ++$i) {
   $t->VariableMethod(1);
}
$Stop = microtime(true);
Row($TestName, $Start, $Stop);

echo '</table>';

Addendum
Upon implementing the method described above, I discovered one caveat to the solution. If you always fill in the full argument list for the method being called, any default values assigned in the method declaration are lost.

For example, if the method I am calling is defined as:

function MyMethod($Arg1 = 'default') {}

And I call it in my __call method as described above:

$this->$MyMethodName($Arg[0], $Arg[1], $Arg[2], $Arg[3], $Arg[4], $Arg[5], $Arg[6], $Arg[7], $Arg[8], $Arg[9]);

The ‘default’ value will be lost and replaced with whatever the value I padded the array with. Obviously this is a big problem. The two solutions are to (a) explicitly enforce the default values within all of your methods [eeek], or (b) count the number of arguments and call the method selectively:

$Count = count($Args);
if ($Count == 0) {
   $this->$Call();
} else if ($Count == 1) {
   $this->$Call($Args[0]);
} else if ($Count == 2) {
   $this->$Call($Args[0], $Args[1]);
} else if ($Count == 3) {
   $this->$Call($Args[0], $Args[1], $Args[2]);
} else if ($Count == 4) {
   $this->$Call($Args[0], $Args[1], $Args[2], $Args[3]);
} else {
   $this->$Call($Args[0], $Args[1], $Args[2], $Args[3], $Args[4]);
}

I ran this kludgey if statement through the same benchmarking code as listed above and found that it took on average 2 – 3 more seconds to run. So, that brings the entire summary of benchmarks up to:

Testing Method Time
Literal Function 1.941
Variable Function 2.792
call_user_func() 5.445
call_user_func_array() 6.376
Literal Method 2.912
__call 6.931
__call with call_user_func_array 17.903
__call with Variable Method 10.727
__call with Variable Method & if statement 12.911

Not as good as I had hoped, but still better than call_user_func_array.

One final note on the subject:
Remember that all of my benchmarks were achieved by calling the methods 2,000,000 (that’s two million) times each. What if I ran another test where the iterations were bumped down to something more like I would expect for one page-load of Garden? Imagine if every single method call on every single object in Garden were run through __call instead of directly to themselves. How many would there be? I’m going to go on the high side and say that there are 200 method calls on a single page-load. And let’s say that the application is pretty busy with 30 people loading pages at the same time; What would the numbers stack up like, then?

200 methods * 30 requests = 6000 calls:

Testing Method Time
Literal Function 0.005
Variable Function 0.008
call_user_func() 0.015
call_user_func_array() 0.019
Literal Method 0.008
__call 0.021
__call with call_user_func_array 0.053
__call with Variable Method 0.033
__call with Variable Method & if statement 0.040

We’re really talking about fractions of a second of difference. Granted, on a heavy-load application that could mean the difference between a slow application and a fast one. But 99% of the people who download and use Garden will *not* be running extremely high-traffic websites where a slowdown would be noticeable, and as Larry Garfield pointed out in his article, you can always throw more CPU at the code to speed it up – and if you’ve got a high profile website, I’m sure that’s just what you’d do.

Personally, I think the benefits far outweigh the micro-seconds in slowdown. I’d be willing to bet that when the application is in alpha, and I start profiling to find bottlenecks, this will not be one of the sore spots.