Effective telediagnostic-platform with rich communicational information in the sensitive situation

2020 – 2022

This project aims to develop an immersive telediagnostic-platform using virtual reality (VR) to safely provide rich information exchange in complex and sensitive dyadic communication situations. The information provider (e.g., patient or prisoner) will wear a virtual reality headset and see a real counsellor (e.g., doctor or lawyer) in virtual space as if meeting in the real world in real-time.

To support rich communicational cue, verbal behaviour, non-verbal behaviour, and the physiological signal will be collected and analysed using a deep learning approach in real-time. Based on the analyse, our platform will provide the expected mental status to the counsellor for better diagnosis. 

In this project, the research team aims to develop a telediagnostic platform, analyse non-verbal cues for rich communication information, and to evaluate our system as an effective consultant in complex dyadic conversation.

This project is funded by a grant from the Science for Technological Innovation National Science Challenge (SfTI) – Kia Kotahi Mai: Te Ao Pūtaiao me Te Ao Hangarau programme hosted by Callaghan Innovation and the Ministry of Business, Innovation and Employment.