Query-by-Gaming: Interactive spatio-temporal querying and retrieval using gaming controller

---

Ramazan S Aygun

University of Alabama in Huntsville, USA

: J Comput Eng Inf Technol

Abstract

---

Inputting spatio-temporal queries is challenging as it requires a sequence of both spatial and temporal information in queries. Despite development of powerful spatio-temporal index structures and querying languages, forming spatio-temporal queries is still difficult for many users. Inspired from games where a user provides a sequence of actions on a spatial layout, we propose the methodology of query-by-gaming for interactive spatio-temporal querying using a gaming controller. We have applied query-by-gaming for spatio-temporal retrieval of content for tennis videos. Queries are built using a gaming controller rather than a mouse or keyboard. First, for a database of tennis videos, the players, ball and shot types of players are indexed with respect to their positions using our semantic sequence state graph (S3G). The user is allowed to select the initial positions of objects on the court and a relevant clip is retrieved from the database to build the query. The user starts with this video clip to select the positions of players and ball including where the ball should be sent after a shot using the controller. The system searches the database and determines if there is a relevant clip and the partial clip is displayed if found. Then the user is permitted to control the other player with an option of skipping events and the query is a built as a sequence of interactions. Meanwhile, the system searches for videos that satisfy the complete query as a conjunction of query segments. At the end of query, the user is provided the list of videos satisfying his or her query. We use linear temporal logic to formally represent our queries. We have also performed user interface evaluation for comparing gamepad-based interface and mouse interface for forming spatio-temporal queries.

Biography

---

Ramazan S Aygun is an Associate Professor in the Computer Science Department at the University of Alabama in Huntsville. He received his BS degree in Computer Science/Engineering from Bilkent University in 1996; MS degree in Computer Engineering from Middle East Technical University, Turkey in 1998; and PhD degree in computer Science and Engineering from University at Buffalo, State University of New York, in 2003.

Email: aygunr@uah.edu