Wizard-of-Oz Recordings

WOZ recordings may be designed in many different settings depending on your needs or the features of the simulated application (`virtual machine'); therefore it is difficult to give detailed instructions on how to set up a WOZ recording site. Please also refer to the section for general hints and tips as they probably apply for WOZ recordings as well. Here are some general hints specially for the WOZ technique that might be useful:

• Do not use a room with a one-way mirror. This technique is so well-known by now that almost every speaker will become suspicious. If you need to watch or video tape the speaker, use small cameras instead, and if you cannot hide them successfully, simply add them to the `virtual machine'<sup>5.9</sup>
• Take care that the location of the wizards (the `control room') is acoustically insulated from the recording room, so that the wizard can produce live speech output and/or communicate (possibly via an intercom) with the recording supervisor.
• Since the microphones are further away from the recording device in a WOZ setting than in a usual setting, make sure that you don't get a hum on your signals. If you experience disturbances, try to set up the microphone amplifier in the recording room and not in the control room.
• In most WOZ recordings you will need a synthetic speech output of the `virtual machine' instead of written feedback. According to our experience it is next to impossible for a wizard to type absolutely error free. On the other hand it is quite easy to distort the human voice - for instance by using sound devices from electronic music stores - and to train the wizards to speak in a very controlled manner. The only weak points in this setting are laughing or coughing (no machine laughs or coughs!).
• Do a lot of training sessions with cooperative, non-naive users and ask them what went wrong. Since WOZ recordings are very expensive because of the manpower you need, it is better to do some extra off-line training sessions before you recruit naive speakers.
• Clarify the legal aspects of recordings in which the recorded person is unaware of the situation. For instance you should ask your legal department whether it is ok not to tell the speakers immediately after the recording that it was a fake. Because if you have to tell them the truth right after the recording, very soon you will not be able to find naive speakers in the vicinity any more.
• Simulating very complex (`intelligent') systems is difficult because the wizard constantly has to decide whether a user input is still within the capabilities of the `virtual machine'. A good way to handle this is to prepare task flow maps. Task flow maps are like a semantic map that shows the wizard which possible ways through a dialog with the `virtual machine' are allowed and which are not. Since the wizard has to think about it (time delay) it is also very helpful to have some pre-recorded standard answer ready that can be played by just pushing a button^5.10. For instance there should be a general ``Sorry-I-did-not-understand-you'' button and probably others that fit exactly to the weak points in the `virtual machine' setup.
• Keep the recording session short and keep the speakers occupied. In long sessions it is more difficult to maintain the illusion that the speaker talks to a machine. Bored speakers start to look around or think about clever ways to trick the system.^5.11

This is by no means a complete list of hints. Many problems you'll find in the pre-tests might be solved by a better or more specific instruction to the speakers as well.