Effective Telediagnostic Platform with Rich Communication in Sensitive Situations



This project aims to develop an immersive telediagnostic platform for remote sessions in situations where there are sensitive relationships, such as counsellor-patient, which require rich communication cues including non-verbal behaviour and physiological signals.

The information provider (e.g., patient) will wear a virtual reality (VR) headset and see a real counsellor (e.g., doctor) in a virtual space as if meeting in the real world. Verbal and non-verbal behaviours, along with physiological signals, will be collected and analysed using a deep learning approach in real time.

The research team aims to develop the telediagnostic platform, analyse non-verbal cues for rich communication information, and to evaluate the system as an effective consultation tool for complex dyadic conversations.

This project is funded by a grant from the Science for Technological Innovation National Science Challenge (SfTI) – Kia Kotahi Mai: Te Ao Pūtaiao me Te Ao Hangarau programme hosted by Callaghan Innovation and the Ministry of Business, Innovation and Employment.




Sungchul Jung

Dilshani Kumarapeli

Simon Hoermann

Rob Lindeman

Thammathip Piumsomboon