The Magic Tool for Archiving and Retrieving Digitised Moving Images Automatically Is Here

Composed by Sten Frykholm, Head of Television Archives, Sveriges Television

Vicar is Esprit project no. 24916

January 27 the final review of the EU Project VICAR took place at ORF in Vienna. Vicar is an innovative search-engine technology for cataloguing and retrieval of huge amounts of video. Vicar stands for Video Indexing, Classification, Annotation and Retrieval. It offers advanced video search and analysis features, including face recognition, camera movement detection and query by example retrieval - based on shape, texture and colour. VICAR has been evaluated by major European broadcasters: SWR (Germany), ORF (Austria), SVT (Sweden) and NAA (Dutch National Audio-visual Archive.

VICAR's design is based on standard PC hardware, client server software and open networking protocols.

At the review the project was passed with flying colours and with lots of praise by the reviewers.

For me, representing one of the User Partners, the final demonstration at the review revealed possibilities that I had not quite seen during the scope of the project. Quite natural as our criticism and demands had forced the Technical partners to developments that were impressive.

I feel that we here really have innovations especially in the search methods. A totally new alternative presentation of the search results "The Stripe Image" gives new possibilities but at the same time new demands of the user. VICAR is not something that just buy and apply, you have to adapt to it and grow into its possibilities, but once you do that you are in business!

The Technical Partners JRS and VCPC from Austria and SMR from The Netherlands have developed several tools within VICAR. The basic tool is the Video Navigator (VIN).

Replacing many routine actions, the Video Navigator automates a part of the cataloguing work that today is still done manually by documentalists. Information is extracted automatically from the video stream in three main steps. First the video is segmented into its single shots. From each shot a number of representative keyframes is selected.

Second, the Video Navigator computes an index for each keyframe. The index is based on a wide variety of features, such as frame colour, distribution of brightness over a frame, camera movements, textures and changes in contrast over a shot. Taken together, the features are integrated into a rich index structure that is linked to the time-code of the original video tape. If a user presents an example image or movie clip, the index will be used to search for similar footage.

In the third step the keyframes are labelled in terms of some class that is relevant to the user. The standard Video Navigator is equipped to classify, recognise and annotate VIPs ('Clinton', 'Kohl'), settings ('interior', 'forest', 'city', 'mountain scenery'), objects ('3 cars') and camera movements ('left', 'pan', 'tilt', 'zoom out/in').

The documentalist retains control over this process; annotations proposed by the system can be overwritten or completed. Semantic interpretations exceeding the bounds of VICAR's knowledge, such as historical significance of events, can be added manually by the documentalist. Finally, it is possible to edit the resulting folder structure representing the shots and keyframes; this functionality enables the documentalist to create a personalised file archive structure to minimise research time (with folders like 'most popular stock shots').

The producer or documentalist can query by text, as usual, and by visual example, using stills or video sequences. In response to a query, the system delivers a set of matching sequences, shots, or frames. Depending on the storage capacity available to the user, the user can play a video instantly in the browser, or retrieve it near-line from a tape store.

In addition, the interface guides the user in a type of query that goes beyond current limits of querying. Given some search results, the user can indicate a number of answers he or she likes best, and run a second query. This kind of relevance feedback enables a user to search without explicitly formulating what he or she is looking for. For instance, a user can search for 'Helmut Kohl' in an interior shot and then select those search results that show him wearing a dark blue jacket against a light blue background. The Video Navigator will now only retrieve clips matching these criteria.

In conclusion, it is possible to control the different functionality's of VICAR's Video Navigator through an intuitive and flexible user interface that is optimally geared towards television archive practice and in particular, promoting reuse of archive material.

For more information contact

SENTIENT MACHINE RESEARCH , Baarsjesweg 224, NL-1058 AA Amsterdam, Netherlands

Attention: Peter van der Putten, pvdputten@smr.nl