There are no shortage of online experiments on the web. Psychological Research on the Web lists hundreds. As the Top Ten Online Psychology Experiments points out, it’s a little hard to assess the validity of these results because of variations in speeed of the hardware. They note that we also don’t know who is taking these tests, or whether they have understood the instructions properly.
The open source stimulus presentation packages for desktops I’ve programmed with (PsyScript, VisionEgg) advertise impressive temporal accuracy for output. (See, for instance the Appendix in Bates & D’Oliviero (2003)). An important question is: how would you verify such assertions for yourself? How would you make sure your experiment software is calibrated properly? When I raised the issue of calibration with my engineer friend Bob Erickson, he suggested that an oscilloscope connected to a light-sensitive diode held up to the screen, similar to what Bates & D’Oliviero describe, would be the best way to check to see whether screen displays last as long as you expect them to.
In the online space, we can’t expect subjects to employ oscilloscopes. When I discussed the issue of non-standard hardware with Jim McGinley, he showed me the video calibration tests used by Rock Band, in which a metronome-like bar swings back and forth, and the user is invited to strum in time with the metronome. The metronome may not be the right approach, but something like this, where the user attempts to hit the space bar at the same time as a phenomenon on the screen, seems like it would be on the right track.
Jim tells me that Flash is supposed to run at 60 frames a second. This means a temporal resolution of 16.66 ms, at least for output, which is plenty good enough for a lot of psychology experiments. This says nothing about input. The real question, however, is how much temporal variability would be introduced by other applications running on the same machine. For some experiments, having a particular stimulus display for a precise number of milliseconds is crucial.
Any thoughts on output calibration, especially for online experiments, would be welcome.
Bates, TC & D’Oliveiro, L. (2003). ‘PsyScript: A Macintosh Application for Scripting Experiments.’ Behaviour Research Methods 35: 565-576.