On-Screen Scientist

My iPhone app OnScreen Pitch Count (submitted to Apple, but not yet approved for App Store placement) is easy to use. Even so, describing how it works, using only words and still images can be tedious and may give the impression that it’s more complicated than it is. The draft user’s guide for OnScreen Pitch Count details what every screen button is for. Most of them should be immediately obvious, but the sheer number of buttons might make learning seem an unpleasant task, at first glance.

I clearly needed a video presentation of OnScreen Pitch Count that showed both how to use the app and what it was to be used for. Not knowing either the timetable or the exact procedure for the Apple review, I also felt the need to get something online as soon as possible, just in case an Apple reviewer, with the app’s fate in his or her hands, came to the web site looking for help in using it. I originally thought that a video of me or someone else tapping the buttons on the screen would be best, as being more realistic and possibly making the viewer want to start doing the same thing. That may be true, but given my resources and my impression of some rather pathetic online videos of iPhone game play, where the player’s fingers really got in the way, and the focus was poor, I decided to opt for making a screen recording of the app running in the iPhone Simulator, which Apple provides for testing apps under development.

Since most button taps lead to the tapped button’s being highlighted, it should be easy for the viewer to follow the action, imagining the invisible finger or thumb of the user. I already had used a Macintosh program iShowU for making a screencast demo of OnScreen Particle Physics, OnScreen Science’s modern physics teaching software, so my first thought was to try iShowU. First I went to its web site to see if there was a new version. There was, one that required Mac OS 10.5. This raised the question of whether the version I had would run under 10.5 at all. Since there was a discounted upgrade price I decided to look into the upgrade. Unfortunately to get to the upgrade page, one had to enter a password, the meaning of which was not explained anywhere I could see. I decided to launch the older version to see if the password might be found there. The good news was that the older version launched OK under 10.5, but the serial number turned out to be extremely long and with no way to copy it. Perhaps some old email would have it, but that was enough to deter me from the upgrade, since I wasn’t even sure the serial number was in fact the required password. I decided to see how well version 1.7.2 would work before revisiting the upgrade or alternative software question.

iShowU provided an easy, intuitive way to capture just the portion of the screen I wanted. There was a draggable, resizable rectangle, which I could put exactly over the iPhone Simulator’s on-screen representation of the iPhone device. The on-screen instructions said to hit return to set the recording area. Nothing happened to indicate that the capture rectangle had been registered when I did that (several times). It seemed this might be an incompatibility with 10.5, as I had feared. What else to do but try quitting and relaunching? That did the trick, though as usual in such cases, I’ll never know why. I did a couple of short practice runs. It was working, saving a QuickTime mov file to the desktop, each time. I then plugged in my USB microphone, the Blue Snowflake, and tested it with commentary enough to see that the audio worked. About a megabyte seemed to be required for each minute of recording, which I thought was within reason.

Clearly, though, I could not just improvise my way all the way through a demonstration. I would have to script it, and try to keep it around five minutes. It was immediately apparent to me that methodically going through the use of each button would be boring, and probably impossible to do without a context for the button usage. At the risk of sounding dumb, I decided to use a made-up game scenario, which I would narrate sparingly, to illustrate the app. I typed the script as fast as I could, thinking of it as a first draft. Looking at it now, I can see a couple of things I might have added, and a couple of places where the order might be improved, but it covers most aspects of the app’s usage to record pitch results in a game. When I do it again, I’ll probably include a runner caught stealing and a batter reaching base after a dropped third strike. I definitely should have shown the use of the Undo button, but its visible presence hopefully makes the point that it is possible to undo actions.

Now, how to do it physically? The script went beyond one page, but I didn’t want to have paper rustling sounds recorded, and I needed to have my hands free to use the trackpad and keyboard on my MacBook Pro anyway. I should note that I turned off the mouse cursor recording feature of iShowU since I didn’t want the movement of a mouse pointer across the screen to be highlighting continually that this was a simulation on a computer screen. I didn’t try to get a visual display of mouse clicks (simulated button taps) since the mouse clicks almost always highlighted the button anyway. One place where the simulation didn’t match the standard iPhone screen was when I typed in names. The iPhone keyboard appeared in the simulation, but I would have had to click with the mouse pointer on each letter in order to show the big letter that appears beside the tapped key. This would have been much too slow, so I just typed on the computer keyboard, which accomplished the job much faster, but without the usual visual display.

I decided to lay the two pages of the script out side by side to avoid the paper handling problem. It was still challenging to go back and forth between the screen and the script, and I lost my place a couple of times and had to do some improvising. The result was bad enough in one six-second interval that I decided to edit the audio track, which I’ll tell about below. I may do more editing later. The method I used for script page display wouldn’t work for anything longer than what I recorded. Maybe I could use a page turner the way pianists do.

Despite the less-than-perfect delivery and the deviation from the script in places, I went ahead and posted the demo as recorded, copying the javascript code I had used previously to play the OnScreen Particle Physics QuickTime demo. One place where my voice trailed off while I obviously searched for what I wanted to say seemed worse and worse the more I thought of it (played it, actually). I wasn’t sure how to edit the audio part of the mov file, but I knew there had to be a way to replace the flawed section. I had Audacity, the free audio recording and editing software distributed under the GNU General Public License, on my hard drive, so I thought I’d give it a try. Audacity imported the mov file via the Open command. I was able to locate the problematic section without any trouble and see that it was six seconds long.

Now what? I had to record a replacement for the bad section. I considered doing it with Audacity, but for some reason I don’t recall now I didn’t succeed immediately, so I decided to use QuickTime Player (Pro) as my recording software. Recording with QuickTime Player proved easy once I realized I had to set the input source as the Blue Snowflake microphone via the Preferences panel, just as with Garage Band. I recorded what I thought was a major improvement both in wording and diction. I then opened the file with Audacity and located the beginning of the substitute sentence. Since it was spoken without hesitation or searching for words, it took a considerably shorter time than the original one had. In order to keep the rest of the audio in synch with the video, I just selected six seconds from the new recording, which included a good bit of silence at the end. I then went to the old mov file audio track, which still had the undesirable six seconds selected, and did a paste. Would it combine the two, insert the new audio before the old, or replace the original as one would expect? It replaced the original. I could now play right through the edited section without hearing any obvious “splicing” effects. I did regret not having given a little silence at the beginning of the new segment, but the result was well within the Guy Kawasaki “Don’t worry, be crappy” standard. I have a feeling anyone that knows there was an edit done somewhere will be able to pick out where it occurred, but it is definitely an improvement over the original.

I still didn’t have the new audio inserted into the video. A straight save from Audacity would only give me an Audacity project file rather than a mov file with a video track, so I did an export to AIFF, an Apple audio format I knew QuickTime could handle. It took a little bit of searching through the QuickTime Player Help, but I found the procedure for adding a soundtrack to a mov file. I opened the new AIFF file I’d exported from Audacity and chose Edit->Select All, then Edit->Copy. Then I opened the mov file with the original sound track and selected Edit->Add to Movie. I played through the “bad” spot and discovered I had just added a second track, so that both versions played in the six-second interval I wanted to change. No big surprise, but I had thought it might just replace the soundtrack. Revert. Then I rediscovered that Window->Show Movie Properties revealed the video and audio tracks of the movie. I selected the audio track and deleted it. Then I repeated the procedure for adding the new sound track and ended up with what I was after. I then replaced the online mov file with the newly edited one. The next edit will be much easier. One of the reasons for writing this is so that I’ll have it for future reference.

So, is a something-less-than-professional, hand-rolled video better than nothing? I have no doubt that it is. There really is no easy way to convey the app’s usage with text and still images. I plan to improve on this first video effort; and, as I await Apple’s approval, I may be moved to do the whole thing over, but I feel it is an asset already. You can see and hear the results here.