SUMMARY

Our project aims to improve user sentiment towards autonomous vehicle (AV) transport through the design and implementation of a simple and effective multimodal interface. Our user group consists of all licensed drivers in the U.S. and our design focus group was mid to late aged car enthusiasts. This project contributes to the research and HCI foundations of the future of AVs in the U.S. and abroad.

DESIGN SCOPE

We designed for a future where fully autonomous vehicles are the standard for transportation. Users have widely adopted the use of AV’s, but some users are apprehensive about making the switch.

What Does a Future of Fully Autonomous Vehicles Look Like?

Vehicles are fully automated. User control is not available. Driving is executed by the AV using an internal process of perception via sensors, and motion planning/motion execution using the AV’s algorithmic processing system.

Here’s what the degrees of automation look like for vehicles based on today’s standards:

A chart comparing various levels of autonomous driving with regard to driver and vehicle role percentages
The first 3 levels of vehicle automation are considered Monitored Driving, and are as follows: Level 0: Driver Only – Eyes On, Hands On The driver is continuously exercising longitudinal and lateral control Level 1: Assisted – Eyes On, Hands On The driver is mainly continuously exercising longitudinal OR lateral control. Minimal lateral or longitudinal control is accomplished by the vehicle. Level 2: Partial Automation – Eyes On, Temporary Hands Off The driver mostly has to monitor the system at all times. The system partially has longitudinal and lateral control in a specific use case. The second 3 levels of vehicle automation are considered Non-Monitored Driving, and are as follows: Level 3: Conditional Automation – Temporary Eyes Off, Temporary Hands Off The driver does not have to monitor the system at all times, but must always be in a position to resume control. The system has longitudinal and lateral control in a specific use case. The system recognizes the performance limits and requests driver to resume control within a sufficient time margin. Level 4: High Automation – Eyes Off, Hands Off The driver is not required during defined uses cases. The system can cope with all situations automatically in a defined use case. Level 5: Full Automation – Eyes Off, Hands Off System can cope with all situations automatically during the entire journey. No driver is required.

 

The AV Control Loop

Autonomous vehicles and their algorithms are currently being processed by a variety of companies. These algorithms are being refined to create an effective AV Control loop. This refers to the process of perception, motion planning, and motion execution. Ansys is at the forefront of this development and their diagram the process:

Chart depicts the AV Control Loop.
Within the simulation’s closed loop, the environment communicates with the vehicle’s sensor models, radar, camera, lidar, ultrasound, speed, GPS, and V2X. These inputs communicate with the AV software, and complete steps of perception, motion planning, and motion execution. The software then communicates to the vehicle components and vehicle dynamics. The vehicle components and dynamics conduct a small subloop which communicates through validation steps, including simulation results, requirements, and scenarios, which then is transformed into data. The data contains models, including the world, vehicle, and software; the physical world and sensors; and the simulation’s world and sensors. This data completes the subloop, and the vehicle components and dynamics then pass information into motion and rendering. The virtual world also takes into account traffic objects and behavior.

By the time our product would be implemented, the algorithm testing phase of AV’s would be stable. Our design would simply convey the information from the AV Control Loop and display it through our interface to our users.

An Oligarchical Industry

Throughout history the automotive industry has resembled an oligarchy. We anticipate this to continue. Our product could be licensed or sold to a large manufacturer for use in their car models.

AV Regulation

Safety and implementation of AVs will be a considerable issue in the future. It is highly likely that regulators could create determining legislation that would dictate requirements of AVs. This could easily apply to our interface in the future. However, we have disregarded any potential onset of restricting regulation. The regulatory future of AVs is largely speculative at this point in time so it is a negligible concern.

PROBLEM

The vast majority of drivers today have at least some level of apprehension towards the idea of riding in a fully autonomous vehicle. The primary reasons for this lack of trust are the driver’s loss of control over the vehicle and overall unfamiliarity of how an AV works.

INTENTION

Our goal is to increase transparency of the functionality of an autonomous vehicle for our primary audience: drivers with previous driving knowledge and experience, with some level of mistrust or hesitancy towards autonomous vehicles, but who may need to rely on them to help maintain mobility due to physical challenges.

SOLUTION

We are creating a multimodal display for inside of a fully autonomous vehicle. Our interface displays the decisions made by the vehicle and the information used to make them.  Our intention is to mirror data a driver would use to make decisions behind the wheel, as well as articulate to the user through multimodal feedback what the decisions are.  This interface will help users feel more comfortable while riding in an AV to increase their confidence in the AV system.

ROLES & RESPONSIBILITIES

This project was conducted by our team and responsibilities were shared equally.  Some tasks were weighted more in alignment to the individual team member’s skillsets.  Overall responsibilities included:

  • User research and persona development
  • Industry and technology research
  • Brainstorming
  • Concept creation + ideation
  • Visual design
  • Prototype development
  • Presentation
TOOLS

Tools used to complete this project include:

  • Google Suite (Docs and Slides)
  • Pen and paper
  • Whiteboard
  • Photoshop
  • Visual Studio (C#)
  • GitHub
  • Azure Web App Service
  • WordPress
PROCESS
  • 1. USER SURVEY

    Before we dove into the project we collected some data from two different interviewees that were both car enthusiasts. Below are some of the questions about autonomous vehicles and their responses.

    1. Have you ever ridden in an autonomous vehicle? Either as a driver or a passenger?
    2. Would you be comfortable riding in an autonomous vehicle?
    3. Why/why not?
    4. Would you ever consider owning an autonomous vehicle?
    5. What information would you want access to in regards to the car’s driving decisions to make you comfortable/more comfortable in an autonomous vehicle?
    6. How would you want to access that information – on dash, in car, auditory, visual, in app, etc.?

    And their responses:

    1. Neither interviewee had ridden in an autonomous vehicle but one had ridden in a Tesla with self-driving features.
    2. Neither interviewee was comfortable riding in an autonomous vehicle.
    3. The interviewee that had ridden in a Tesla had a bad experience and the other doesn’t trust the computer to make the right judgement calls.
    4. Neither interviewee would consider owning an autonomous vehicle at this point.
    5. Both interviewees would want to know as much as possible about the vehicles designs including its decisions, information on the vehicle’s status, and an explanation of the decisions.
    6. One interviewee preferred the traditional dashboard while the other preferred an app or website portal.
  • 2. COMPETITIVE ANALYSIS

    As of the end of 2021, there are no fully autonomous vehicles at this time. The closest thing is Tesla’s vehicles with self driving assistance. At this point all vehicles require human input and can accept human overrides of the vehicle’s decisions. Fully autonomous vehicles would have no human input at all.

  • 3. PERSONAS

    From the survey information we decided to focus on the early adopters of this product since fully autonomous cars are not available in the market as of yet. We did focus down on a subset of users that would have trouble using a traditional vehicle due to health conditions as this would give them a way to travel without driving themselves.

    Phil
    A middle-age man wearing a t-shirt, shorts, sneakers, and a baseball cap leans with arms crossed on a silver sports car in a suburban neighborhood.
    • Name: Phil
    • Age: 52
    • Occupation: Bank manager

    Bio: Phil is a bank manager who loves cars and traveling. Phil’s biggest hobbies involve some sort of traveling whether it is going to a car show, going to the movies, or going to a nearby park to do some hiking.

    Attitude: independent, hard working

    Behavior: family oriented, travel enthusiast, worried about losing independence

  • 4. PROTOTYPING

    Our initial prototype consisted of a series of photos mapping out the general concept of the interface. This layout served as the foundation for our final, interactable prototype.

    A dashboard mockup of figure 1, depicting the user interface. Features include destination, arrival time, date, time, temperature, system messaging, road conditions, speed, acceleration, road position, traffic, fuel gauge, time to next fuel-up, odometer, speed limit, overview of system status, and toggle to admin controls.
    Figure 1: General layout as may be seen while the vehicle is in operation.

    A dashboard mockup of figure 2, depicting the user interface with all possible icons displayed. This includes all features depicted in Figure 1, as well as risk avoidance messaging, turn signals, steering wheel turning, and error lights, such as seatbelts, check engine, hazards, and airbags.
    Figure 2: All icons within the interface are displayed.

    A dashboard mockup of figure 3, which is a close-up of the system messaging panel in the larger interface. Example messages include: "Brakes Applied - Side Collision Avoided", "Next Turn in 1 Mile", "Minor Sensor Fail - Tap for More Info", and "Critical Sensor Fail - Tap for More Info".
    Figure 3: Example messages the system may communicate to the rider(s).

    A dashboard mockup of figure 4, which depicts the system status screen. This includes an aerial view of the car with dots to indicate sensor placement and status. Individual system messaging and metrics depicted include Engine RPMs, Air Intake, Coolant, Battery Voltage, Tire Pressure, Engine Oil Life, Washer Fluid Level, and Brake Life. An overall message indicates that All Systems Are Functioning Normally. A toggle button back to the main screen is also included.
    Figure 4: System status screen. Displays the status of various sensors and under-the-hood components.

    Our final prototype, created with Azure, kept this layout and added interactivity. The user can set their destination and toggle between the main screen and the system status screen. The dashboard indicators can be selected and the meaning of the icon will be shown. A walkthrough of the mockup can be seen in the Storyboards section below.

  • 5. STORYBOARDS

    Phil decides to spend his day off hiking at Acadia National Park. He starts his autonomous vehicle, the dashboard displays, and he enters his destination:

    An animation of the user interface depicts Phil entering his destination of Acadia National Park.

    Once he is on the highway, he wants to make sure that his vehicle is functioning properly. He checks the main dash indicators in the bottom left corner of the interface, and then navigates to the System Status screen to check the status of other components:

    An animation of the user interface depicts Phil checking his system status through the icons on the main screen, as well as toggling to the system status screen for additional details on vehicle health.

    Everything seems to be in order! Phil is reassured that the vehicle is in a safe condition to operate.

    A few minutes later, the driver of a non-autonomous vehicle fails to check his blind spot, which Phil’s vehicle is in, and attempts to merge into Phil’s lane. The sensors of the autonomous vehicle detect that a side collision will occur if it does not slow down and allow the other driver to merge in front of it. The brakes are rapidly applied and, fortunately, enough distance is created from the other vehicle to avoid an accident:

    An animation of the user interface depicts Phil's vehicle braking to avoid a side collision. A system message stating "Brakes Applied - Side Collision Avoided" is displayed.

    To Phil’s relief, no other close calls occur during his transit. The vehicle safely arrives to the parking lot of Acadia and alerts Phil that he is now at his destination:

    An animation of the user interface depicts Phil arriving at his destination. A system message stating "Arrived at Destination" is displayed.

  • 6. IMPLEMENTATIONS

    We could not make a fully implemented solution as we don’t have a vehicle that has all the sensors. We implemented a mockup of what the dashboard would be with options for the user to control the display. The finished product would be controlled by sensors instead of a user.

    The mock up website is built using a randomizer that will update the UI.

    The mock up does have an option to be installed as an app to support offline use as the autonomous car may not always be connected to the internet.

    System messages are read to the user when added by the sensors or in this case the randomizer.

    Help text are added for logos so that anyone can know what it means easily.

    There is a spot for the user to input the destination so the car knows where to go. This would be the only input the user would use in the finished product.

    Multimodel Autonomus Vehicle Dashboard

CONCLUSION

Our design is part of an evolving industry of competitors who are all investigating ways to improve the AV landscape. With our goal of creating comfort for apprehensive users in mind we have attempted to replicate as much information from a traditional driving interface. This creates familiarity for users and acts as a foundation for us to integrate autonomous decision making information.

AV’s Today
Tesla is at the forefront of technological innovation for AV function. In 2019, they released a demo of one of their vehicles executing high autonomous driving. This is one step closer to full autonomy and the most advanced version of an AV that is currently available. See their demo here:

The Tesla AV interface captures autonomous function on the left of their interface and displays navigation on the right. This design caters to users with a comfortable background in technology. Traditional driver information is displayed, but not emphasized. In this example driver controls are available so the interface was not designed for a fully autonomous vehicle.

Focusing on the Apprehensive User

Tesla’s implementation does a great job capturing and displaying autonomous function. Our design brings AV function to the locus of the interface so users can constantly synthesize the interface display with their own perception of their riding experience. Surrounding this central feature of our interface is traditional driver information as explained in our storyboard and prototyping.

Apprehensive users benefit from our design because it provides them information as if they were in control of the car. We mitigate the lack of driver control by displaying the AV function at the focal point of our UI. This implementation gives users a comfort basis that allows for their confidence as AV passengers to grow as they ride.

Future Implementation

The design we have established thus far is still in a rudimentary state compared to what will eventually hit the AV market in the future. Our core design has created the foundation for an interface that suits the needs of apprehensive users. Future work could include expansions from the primary UI so users could view more specific information regarding vehicle conditions, environment conditions, status of the AV function, and other supplemental information. Beyond this, one of the largest barriers to market entry is a extensive amount of testing that would be required to harmonize our interface functionality with the vehicle itself.