An Automated End-to-End Lecture Capturing and …

1 An Automated End-to-End Lecture Capturing andBroadcasting SystemCha Zhang, Jim Crawford, Yong Rui and Li-wei HeMicrosoft Research, One Microsoft Way, Redmond, WA 98052, USA{chazhang, jamescr, yongrui, present a complete End-to-End system that is fully auto-mated and supports Capturing , broadcasting , viewing, archiv-ing and search. Specifically, we describe a system architec-ture that minimizes the pre- and post-production time, anda fully Automated Lecture Capturing system callediCam2,which synchronously captures all the contents of the Lecture ,including audio, video and visual aids. As no staff is neededduring the Capturing and broadcasting process, the opera-tion cost of our system is negligible. The system has beenused on a daily basis for more than 4 years, during which467 lectures were captured with 17,000+ online and Subject [Informa-tion Systems Applications]: GeneralGeneral Terms:Algorithms, DesignKeywords: Automated Lecture Capturing , live/on-demandbroadcasting1.}

2 INTRODUCTIONLive/on-demand Internet broadcasting of lectures in work-place, conferences and educational settings has attractedmore and more interest recently due to the improvementsin network bandwidth, computer performance and compres-sion technologies. Although online viewing provides a con-venient way for people to view lectures at more comfort-able time and location, the cost of providing such servicescan be prohibitive, primarily due to the cost of hiring pro-fessional videographers and pre-/post-production there have been a lot of efforts in building auto-mated/intelligent Capturing systems, such as the AutoAudi-torium by Bianchi [3], the Classroom 2000 project [1] atGeorgia Tech, the Cornell Lecture Browser by Mukhopad-hyay and Smith [4], the Berkeley Internet broadcasting Sys-tem (BIBS) [5] at UC Berkeley, and the ePresence system [2]at University of Toronto.

3 Some of them focused on auto-mated content production [3, 1, 4]; while others focusedon Automated broadcasting [5, 2]. Unfortunately, none ofthe existing systems have provided a complete end-to-endCopyright is held by the 05,November 6 11, 2005, 1-59593-044-2/05 DeviceSlideCapture DeviceScriptAV contentAV, scriptScript, administrative infoControl infoSlide imagesSlide imagesAV, scriptcopyFigure 1: system architecture of our that automates both Lecture Capturing and live/on-demand paper presents a complete End-to-End system thatis fully Automated and supports Capturing , broadcasting ,viewing, archiving and search. At the architecture level, wehave designed the system such that minimum pre- or post-production is needed. As soon as the speaker walks intothe Lecture room and gets ready for his/her presentation, wemay start the Capturing and live broadcasting process with asingle click.

4 Remote users can immediately see what is hap-pening in the Lecture room. Shortly after the Lecture ends(less than one minute), the recorded Lecture will be avail-able for on-demand viewing, where the remote users havethe flexibility to flip through the slides and re-synchronizeaudio/video with slides or vise versa. The contents of thelecture, including audio, video and visual aids such as slides,are captured with our Automated Lecture Capturing developed based on our previousiCamsystem [6], and focuses on portability while improvingits overall performance. The system captures high qualityaudio and generates professional-looking edited videos byswitching between multiple camera views according to cine-matographic rules. Visual aids are synchronously capturedfor easy browsing and re-synchronization during on-demandviewing.

5 The whole Capturing and broadcasting system wasdeployed in June, 2001. By the end of April, 2005, it hadcaptured 467 lectures with 17,278 online system ARCHITECTUREF igure 1 shows the architecture diagram of our systemwhen Capturing a Lecture . TheiCam2automated audio/video(AV) device and the slide capture device are the contentproviders, which will be discussed in detail in Section 4. Theproduction console plays a central role, which coordinatesthe AV and slide Capturing processes. When a slide change808 Figure 2: Live viewing web detected, the production console sends commands to theslide encoder to trigger the Capturing of slide. At the sametime, it embeds a script command to the AV stream, whichtells the remote client to update the slide image at that in-stance. Hence the AV stream and the slides are always syn-chronized on-the-fly.

6 The AV encoder encodes both the AVstream and the script information into a single live stream,and sends it to the streaming server for live slide encoder compresses slide images and sends themto a storage server, which is also accessed during live prepare for on-demand viewing, a copy of the AV/scriptstream is sent from the streaming server to the storage serverduring the Capturing process. The production console alsoduplicates the script information to the database allows the remote client to watch the presentation attheir own pace during on-demand viewing, because the scriptson the database server can be accessed at any order. Withthese efforts during the process, the Lecture will be immedi-ately available for on-demand viewing after the THE WEB INTERFACEF igure 2 shows the web interfaces for Lecture viewing onthe remote client.

7 The video steam is displayed on the left,and the slides are on the right. If the Lecture is live, theuser can watch the current video and slides synchronously,similar to watching a live event broadcast by a TV the Lecture was recorded before, the user will have moreflexibility. He/she can browse all the slides used in the talk(bottom window of Figure 2) by hitting the All clicking a thumbnail slide image will show the cor-responding slide in the slide region. The user can also movethe slider of the embedded Windows Media Player and jumpto any place in the AV stream. The Sync V button allowsaudio/video to synchronize with slides, and the Sync S button synchronizes slides to Automated Lecture AudioWe use a wireless clip-on microphone to capture the speaker,and an 8-element microphone array to capture the ques-tions raised from the audience.

8 The processed audience andspeaker audio signals are then digitally mixed via an in-telligent audio mixer, which suppress the microphone arrayaudio when the speaker is Visual aidsSince we do not acquire the original visual aids from thespeaker before the Lecture , we capture them on-the-fly. Oursystem can capture both slides and are scanned by a WolfVision VZ-57 visualizer , which has VGA output. A distribution amplifierselects one of the inputs from the visualizer or a laptop,and sends it to the projector. Meanwhile, a copy of thedisplay is digitized by a video capture card. Automatic slidechange detection is enabled which makes use of the differencebetween neighboring VideoIniCam2, we use two network cameras a speaker cam-era and an audience camera. Network cameras allow us toplace the capture and broadcasting servers anywhere on theInternet, which greatly improves the system s speaker camera will generate overview of the lectureroom, speaker view, and screen view.

9 The audience cam-era will produce overview of the audience, and questionerview pointing directly to the audience who asks a camera feeds its video to a virtual cameraman (VC).These VCs send their videos to a central virtual director(VD), which selects one of the inputs as the output video processing components, including the VCs andthe VD, are wrapped in a video source Direct Show filter,which can be used as a regular video device. In additionto the hardware upgrade, iniCam2, a hybrid tracking algo-rithm is developed to track the speaker with a single PTZcamera. The cinematography rules are now editable usinga scripting language, which makes it possible to port oursystem to various room CONCLUSIONSWe have presented a fully Automated End-to-End lecturecapturing and broadcasting system . The contributions ofthis work are two-fold.

10 First, we proposed a system archi-tecture that minimizes the pre- and post-production allows remote clients to view lectures both live and on-demand. Second, we reported our most recent progress onautomated speaker/audience Capturing , namely theiCam2system. The result is a system that Automated both lecturecapturing and Lecture broadcasting , making the operationcost of the system REFERENCES[1] G. Abowd. Classroom 2000: an experiment with theinstrumentation of a living educational Journal, 38(4):508 530, 1993.[2] R. Baecker. A principled disign for scalable internet visualcommunications with rich media, interactivity andstructured Centre for Advanced Studies onCollaborative Research, 2003.[3] M. Bianchi. Autoauditorium: a fully automatic,multi-camera system to televise auditorium Joint DARPA/NIST Smart Spaces TechnologyWorkshop, 1998.