Building Multimodal Web AI Agent

- We first build MultipanelVQA benchmark to challenge Large Vision-Language Models with their ability to understand multipanel images, such as web screenshot and posters.
- We are now working on developing specialized AI agent to interact with all kinds of UI, including web, mobile, etc.

Respond to Help Requests (R2H) Project (EMNLP 2023)

- We establish the R2H benchmark, featuring tasks that assess an agent's capabilities based on guiding users or another agent in unknown areas through dialogues.
- We propose two multimodal navigation-helper agents, fine-tuned SeeRee model for multi-modal response generation and employing a large language model in a zero-shot manner, analyzed via benchmarking and human evaluations.

Project page

Amazon Alexa Prize competition: Socialbot Grand Challenge 5

- The challenge aims at advancing conversational AI. University teams are tasked with developing a "socialbot", an AI chatbot that can interact naturally and intelligently with humans on a variety of topics through Amazon's Alexa platform.
- I serve as the team leader of our Athena3 team.
- Our Athena team has secured the second-place in the scientific innovation category of Alexa Prize SocialBot Grand Challenge 5.

Alexa Prize Socialbot Grand Challenge

Amazon Alexa Prize competition: Simbot Challenge

- The challenge is focused on helping advance development of next-generation virtual assistants that will assist humans in completing real-world tasks by continuously learning, and gaining the ability to perform commonsense reasoning.
- Our SlugJARVIS Team won the third place in the Simbot Challenge.
- Our SlugJARVIS Team won the Public Benchmark Challenge.

Alexa Prize SimBot Challenge Public Benchmark Challenge

Aerial Vision-and-Dialog Navigation (AVDN) Project (ACL 2023)

- Our AVDN project aims at building drones that understand and follow natural language commands, facilitating hands-free control and accessibility.
- We build AVDN dataset of over 3k recorded dialogs and navigation trajectories and drone simulator with a photorealistic environment.
- We successfully host public AVDN Challenge at the ICCV 2023 CLVL workshop.

Project page AVDN Challenge

Unspuervised Adrenomyeloneuropathy disease data analysis

- Apply feature selection to find dominant factors among the disease progression.
- Design the extra-data-dimension heatmap toolkit for visualization the patient clusters. - Use Bayesian Neural Network to classify the progressor with uncertainty.

Heatmap Toolkit

Learn by Observation: Imitation Learning for Drone Patrol from Raw Videos of A Human Navigator (IROS 2020)

- Design a data auto-labeling method using inter-frame geometric consistency.
- Bring up a DNN called UAVPatrolNet for Detecting Road.
- Make a dataset for drone autonomous Navigation.

Project page

Training Quadruped Robot with Reinforcement Learning

- Use Unity3D ML-Agents.
- Train the quadruped robot to crawl.

Object Detection in Aerial Image
- I contributed to the teamwork by reproducing existing mature algorithms, e.g. RPN, Faster R-CNN.
- I conducted simulated experiments and adjusted the parameters to realize the optimal training effect; improved the object detection performance on aerial images.

ECCV Workshop - Visdrone2018

Control and Monitoring System of DJI Drones through PC
- Designed the control interface on PC with varies functions like “vehicle detection”.
- Developed a system to transmit data between UVA and PC using Qt and DJI SDK.
- Applied the system in city traffic to successfully improve the management efficiency.

Github

Control of Carbon-free Car
- Developed a circuit board and selected the proper sensor by studying the control system of the carbon-free car.
- Conducted OOP of the machine by designing and applying the control algorithm.
- Awarded the First Prize in Engineering Training Integration Ability Competition of Shandong Province.

Assembling and Programming Drones with ROS
- Assembled drones from scratch.
- Designed programs for STM32 flight controller(Pixhawk) running Robot operating system.
- Won the municipal First Prize in China RoboWork Competition.

Github