post

CAT – Continuous Annotation Tool 

Introduction

The continuous annotation tool (CAT) is a standalone software application for continuously rating experimental video material. The CAT software is designed specifically for continuous annotation of EMOTE-like human-robot interactions. It provides the annotator with three different views i.e., frontal, lateral and top-down, which are automatically synchronized for playback (actual synchronization is dependent on the initial recordings). Ratings are input using the thumbstick of a XBOX-like gamepad, whilst familiar actions such as play, pause and stop are possible using either the gamepad buttons, keyboard or mouse. The ability to pause the rating process at any point is also further supported by the option to amend input for the previous five seconds, this allows the annotator to instantaneously correct any mistakes.

How CAT was used in EMOTE

The simplest method of annotating video material is to use relative ratings and discrete segments of video media. However, in EMOTE we wanted data in a far higher resolution, giving us the flexibility to use the final computed ‘ground truth’ for various different purposes (i.e., statistical analysis and training machine learning algorithms). Therefore, it was specified in advance, that the result of these annotations would be a continuous measure, logged throughout the duration of each interaction, for each objective criteria. For example, in EMOTE we annotated the video material for valence, arousal, and task and social attention.

Off-the-shelf software could not provide us with the flexibility we needed to perform the annotations, nor could it deal with the type of input modality which we had in mind, so we opted to develop a bespoke application (see Figure 1) which we have named ‘CAT’ (for Continuous Annotation Tool). CAT was purposely designed to facilitate all of our requirements, such as synchronously displaying three different views of the interaction i.e., frontal, lateral and top-down, and providing a simple visual representation of the rating intensity (in real-time) on a vertical slider bar, which transitioned from green at the very top to represent a positive or high intensity, orange in the centre to represent a neutral intensity and red at the bottom to represent an extremely low intensity.

cat1

Figure 1- Screenshot of the CAT software in action

Ratings are automatically logged with two decimal point precision, with maximum and minimum extremes of 1 and -1 respectively. CAT also allows annotators to pause, restart and amend ratings (for the previous several seconds) at any point in the process, which is useful for fixing any accidental or incorrect input, further ensuring that the outputted signals are an accurate representation of the objective criteria.

Another incentive for creating a bespoke application was the flexibility for testing various input modalities. We wanted to explore the use of a game-pad, or more specifically the thumb stick of a game-pad. Releasing the thumb stick returns the intensity of the rating back to neutral extremely quickly, whereas a mouse or keyboard would cause periods of uncertainty in the outputted signal, due to the fact that the active effort and time required to return the slider back to neutral would be recorded within the signal, furthermore these later modalities are unable to offer the same fine grained resolution. We found that the thumb-stick was also quite comfortable during the long periods of time in which the annotators were required to watch each recording.

Download and Setup Instructions

The source code and binaries for the CAT software can be found here:

http://gaips.inesc-id.pt/emote/?post_type=clink&p=2618 

Basic Setup:

This application has been tested and works on windows 7, 8 and 8.1.

(1) DirectX 9.0

Before you can use this application, you need to ensure that the PC your using has a valid install of DirectX 9.0. Many computers will already have this installed, but in any case you should run the provided installer to be sure you have the same version.

  • Browse to the folder you unzipped everything to and double click “DXSETUP”.
    1. Accept the terms and conditions
    2. Follow the on-screen instructions

Direct X should now be installed!

(2) Application Install

  • Browse to the folder where you unzipped everything and double click “CA_SETUP”.
    1. Follow the onscreen instructions.

The application will install itself into windows and can always be uninstalled via the usual Add and Remove Programs windows application (if needed). The program will run automatically and also add an application icon to the start-> apps window for next time.

Key Points

  • You can add the objective criteria to the application so it is visible on screen.
  • Log files are stored in the c:\WOZAnnotationLogs in a sub folder which relates to the coders name.
  • Logs automatically export at the end of a video. So make sure you watch the video to the end.
  • The stop button has been removed and a restart media button has been added instead. Restarting the media will also reset the logs so make sure you export anything you have done first.
  • Log files are incremented in name, so it doesn’t matter if one already exists or if it is open.
  • Log files are now separated with tabs.
  • Logging manager is more descriptive.
  • Previously video files would play a little so you could see when they have loaded, but this has been removed and now you are informed at the top of the screen…
  • General improvements…

The first screen you will see is the settings form. This information you put in this form allows for persistence, so it will be saved for next time. The settings file is stored in your c:\WOZAnnotationLogs folder.

cat2

The information you place in this form will also be saved into the log files. So you can refer back to them for more info.

 

(1) Connecting a Gamepad: (This is now done automatically when you run the application)

cat3

Connect a suitable gamepad to the computer. Once the application locates the gamepad the gamepad icon at the top right hand side of the screen will turn from red to green. The gamepad should have a button on it named analogue, press this button to ensure you can get values between -1 and 1.

2) Open a Video:

cat4

If you roll your mouse over the top left hand video area the playback controls will appear. They will hide again when you move the mouse away from this area. If you press the icon which looks like a folder, you can browse to where you are storing the WOZ videos.

Browse to the video folder for your chosen participant and select just one of the videos and then click Open. The videos may take a second or two to open, but once this is done you will notice that the camera icon at the top right corner of the screen will turn green to indicate when the videos have loaded. In the Logging Manger box you will notice that the filename of the video media is added automatically, this will save you having to remember and type the name of each log file.

(3) Press Play:

When you play a video file, the logging manager will start automatically. You can see the cat5status change from idle to logging.   The actual log file will not be exported until you request it. You can always add prefixes and suffixes to the filename if you produce more than one for the same participant. The log file will be exported with the name provided in the filename box to C:\WOZAnnotationLogs\”codername”\xxx.txt.

cat6

(4) Continuously Annotate

 

You can annotate videos using a measure of your choice. It’s up to you! Originally the application was designed to evaluate both social and task engagement, but realistically you can evaluate any variable you choose…

cat7In the centre of the application window you will find a large vertical slider (scale) with two small buttons positioned underneath (“L” and “R”), these buttons give you the option of using either the left or the right thumbstick on the gamepad.

If you move the selected thumbstick up and down you will see the scale move.

Additionally, during media playback you will also see the 5 second histogram moving in line with your scale. This doesn’t just provide a visual representation of your evaluation measure. It allows you to amend any mistakes made in the previous five seconds. To do this you should pause the video using F6 and then change the positions of the mini sliders to suit you. Just make sure you save any changes…

 

 

 

 

Disclaimer:

You should trial the application before you fully rely on it. For example, test it by opening and playing videos and test that your Gamepad connects properly. Also, perform actual annotation tests and check the logs are correct before using this software.