US Pat. No. 10,401,852

TELEOPERATION SYSTEM AND METHOD FOR TRAJECTORY MODIFICATION OF AUTONOMOUS VEHICLES

Zoox, Inc., Foster City,...

1. An autonomous vehicle comprising:one or more processors;
a communication interface, the communication interface configured to establish a communications link between the autonomous vehicle and one or more other devices or autonomous vehicles;
one or more sensors; and
memory having stored thereon processor-executable instructions that, when executed by the one or more processors, configure the autonomous vehicle to perform operations comprising:
obtaining sensor data from at least one of the one or more sensors;
determining, based at least in part on the sensor data, that an event has occurred;
determining, based at least in part on the event, a probability that the autonomous vehicle can continue to navigate to a destination;
determining, an event type of the event;
determining an available bandwidth of the communications link;
transmitting, via the communication interface, based at least in part on the probability being equal to or below a threshold probability and based at least in part on the available bandwidth, a request to a teleoperator, the request including the event type and a representation of the sensor data;
receiving, via the communication interface, a teleoperator command, wherein the teleoperator command is based at least in part on a simulator suggestion, the simulator suggestion indicating that a corresponding teleoperator suggestion run on a simulator mitigates the event, the simulator including a simulated autonomous vehicle encountering a substantially similar representation of the event in a synthetic environment; and
controlling, based at least in part on the teleoperator command, the autonomous vehicle.

US Pat. No. 9,734,455

AUTOMATED EXTRACTION OF SEMANTIC INFORMATION TO ENHANCE INCREMENTAL MAPPING MODIFICATIONS FOR ROBOTIC VEHICLES

Zoox, Inc., Menlo Park, ...

1. A method, comprising:
receiving, at a computing system, first data sensed at a first time by a sensor system of an autonomous vehicle, the first
data being representative of a first object of a plurality of objects on or proximate a road surface in a region in an environment
the autonomous vehicle has autonomously navigated;

comparing the first data with reference data associated with a plurality of reference semantic classifications;
based at least in part on the comparing, determining that the first object does not match any of the plurality of reference
semantic classifications;

identifying additional objects having additional object data similar to the first data, the first object and the additional
objects comprising a subset of the objects;

receiving, at the computing system, second data sensed subsequent to the first time, the second data representing a behavior
of at least one of the additional objects;

based at least in part on determining that the first object does not match any of the plurality of reference semantic classifications,
determining, based on the first data and the second data, a probability that the subset of the objects conforms to a behavior;

generating, at the computing system, an inferred semantic classification associated with the subset of the objects when the
probability indicates a pattern of objects in the subset of the objects conforming to the behavior;

associating the inferred semantic classification with the plurality of reference semantic classifications, the inferred semantic
classification being different from each of the plurality of reference semantic classifications;

updating, at the computing system, map data associated with the environment to include information about the inferred semantic
classification; and

transmitting the updated map data to the autonomous vehicle and at least one additional autonomous vehicle.

US Pat. No. 9,754,490

SOFTWARE APPLICATION TO REQUEST AND CONTROL AN AUTONOMOUS VEHICLE SERVICE

Zoox, Inc., Menlo Park, ...

1. A method comprising:
receiving, from a user device, a ride request to transport a user to a destination from an origin location through an autonomous
vehicle system service;

determining, for each of a plurality of substantially identical autonomous vehicle systems associated with the autonomous
vehicle system service, an operational efficiency metric, the operational efficiency metric being based at least in part on
at least one of a sensor calibration of one or more sensors of the autonomous vehicle system or a data storage capacity level
of the autonomous vehicle system;

determining, based at least in part on the operational efficiency metric and at least in part on the origin location associated
with the request, an autonomous vehicle system from the substantially identical autonomous vehicle systems;

controlling the autonomous vehicle system to proceed to the origin location; and
providing information about the autonomous vehicle system to the user device.

US Pat. No. 9,494,940

QUADRANT CONFIGURATION OF ROBOTIC VEHICLES

Zoox, Inc., Menlo Park, ...

1. An apparatus, comprising:
a structural section including a propulsion unit, a steering unit and a sensor unit;
an autonomous vehicle including a plurality of the structural sections mechanically coupled to each other; and
an autonomous vehicle controller in electrical communication with the propulsion unit, the steering unit, and the sensor unit
in each structural section.

US Pat. No. 9,507,346

TELEOPERATION SYSTEM AND METHOD FOR TRAJECTORY MODIFICATION OF AUTONOMOUS VEHICLES

Zoox, Inc., Menlo Park, ...

1. A method comprising:
receiving a teleoperation message via a communication link from an autonomous driverless vehicle;
detecting data from the teleoperation message specifying an event associated with the autonomous driverless vehicle;
identifying one or more courses of action to perform responsive to detecting the data specifying the event;
calculating corresponding ranks for the one or more courses of action; simulating the one or more courses of action to produce
one or more simulated courses of action; calculating corresponding simulation values for the one or more simulated courses
of action;

generating visualization data to present information associated with the event and at least a subset of the one or more courses
of action in association with the corresponding ranks and simulation values to a display of a teleoperator computing device;

receiving data representing selection of a selected course of action from among the one or more courses of action presented
on the display; and

transmitting the selected course of action to the autonomous driverless vehicle.

US Pat. No. 10,133,275

TRAJECTORY GENERATION USING TEMPORAL LOGIC AND TREE SEARCH

Zoox, Inc., Foster City,...

1. A system for implementing a control algorithm for an autonomous vehicle, the system comprising:one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive one or more symbols, the one or more symbols including at least one static symbol or at least one dynamic symbol;
determine, based at least in part on the one or more symbols, one or more features;
determine, based at least in part on the one or more symbols or the one or more features, one or more predicates;
determine, based at least in part on the one or more symbols, one or more features, or one or more predicates, one or more linear temporal logic (LTL) formulas;
determine, based at least in part on the one or more LTL formulas, one or more automaton;
utilize a Monte Carlo Tree Search (MCTS) to generate one or more candidate trajectories;
evaluate the one or more candidate trajectories using the one or more automaton, wherein the one or more automaton verifies that the one or more candidate trajectories satisfies the one or more LTL formulas associated with the one or more automaton;
determine, based at least in part on a cost function, a cost associated with a trajectory of the one or more candidate trajectories;
select, as a selected trajectory, the trajectory of the one or more candidate trajectories based at least in part on the cost; and
control the autonomous vehicle based at least in part on the selected trajectory.

US Pat. No. 10,048,683

MACHINE LEARNING SYSTEMS AND TECHNIQUES TO OPTIMIZE TELEOPERATION AND/OR PLANNER DECISIONS

Zoox, Inc., Foster City,...

1. An autonomous vehicle comprising:one or more processors;
a communication interface, the communication interface configured to establish a communications link between the autonomous vehicle and one or more other devices or autonomous vehicles;
one or more sensors; and
memory having stored thereon processor-executable instructions that, when executed by the one or more processors, configure the autonomous vehicle to perform operations comprising:
obtaining sensor data from the one or more sensors;
detecting, from the sensor data, an event, the event being associated with operation of the autonomous vehicle;
determining a type of the event;
requesting, via the communication interface, a teleoperator command based, at least in part, on the type of the event;
receiving via the communication interface a teleoperator command; and
controlling the autonomous vehicle based, at least in part, on the teleoperator command.

US Pat. No. 9,939,817

INTERNAL SAFETY SYSTEMS FOR ROBOTIC VEHICLES

Zoox, Inc., Menlo Park, ...

1. A system, comprising:sensors operative to generate sensor data for an autonomous vehicle located within an environment, wherein at least one of the sensors includes a LIDAR sensor;
safety systems, including one or more interior safety systems, one or more exterior safety systems, and one or more drive systems; and
one or more processors configured to perform actions, including:
determining, based at least in part on a first portion of the sensor data, a location of the autonomous vehicle within the environment, wherein the location of the autonomous vehicle identifies one or more of a position or an orientation of the autonomous vehicle within the environment;
calculating, based at least in part on the location of the autonomous vehicle and at least a second portion of the sensor data, a trajectory of the autonomous vehicle, wherein the trajectory indicates a portion of a planned path associated with navigating the autonomous vehicle between at least a first location and a second location within the environment;
identifying, based at least in part on a third portion of the sensor data, an object within the environment;
determining a location of the object, wherein the location of the object identifies one or more of a position or an orientation of the object within the environment;
calculating, based at least in part on the location of the object, a trajectory of the object;
determining a threshold distance based at least in part on the trajectory of the autonomous vehicle and the trajectory of the object;
selecting a safety system to activate based at least in part on one or more of the object, the trajectory of the object, or the threshold distance; and
causing the safety system to be activated.

US Pat. No. 9,630,619

ROBOTIC VEHICLE ACTIVE SAFETY SYSTEMS AND METHODS

Zoox, Inc., Menlo Park, ...

1. A method comprising:
receiving trajectory data representing a trajectory of a driverless, autonomous vehicle in an environment external to the
autonomous vehicle;

sensing an object in the environment;
determining object data comprising a position of the object, an object type of the object, and an object classification of
the object;

determining one or more predicted locations of the object in the environment based on the object data;
determining a predicted collision location based on the trajectory data and the one or more predicted locations of the object
in the environment;

determining a first threshold location in the environment between the object and the predicted collision location;
determining a second threshold location in the environment between the object and the predicted collision location, the second
threshold location being closer to the collision location than the first threshold location;

sensing an updated position of the object in the environment;
implementing a first safety action using a safety system in response to the updated position corresponding to the first threshold
location; and

implementing a second safety action using the safety system in response to the updated position corresponding to the second
threshold location.

US Pat. No. 9,958,864

COORDINATION OF DISPATCHING AND MAINTAINING FLEET OF AUTONOMOUS VEHICLES

Zoox, Inc., Menlo Park, ...

1. A method comprising:determining a destination location for an autonomous vehicle of a plurality of autonomous vehicles;
determining a plurality of origination locations to which the plurality of autonomous vehicles are to be directed;
maintaining policy data for selecting individual autonomous vehicles of the plurality of autonomous vehicles to service respective requests associated with the plurality of origination locations, the policy data including one or more of:
a distance that the individual autonomous vehicles are instructed to maintain from external objects;
one or more types of road lines that the individual autonomous vehicles are instructed not to cross;
one or more lanes in which the individual autonomous vehicles are instructed to travel; or
one or more road segments or intersections that the individual autonomous vehicles are instructed to bypass;
generating routing data that indicates at least a route from the destination location of the autonomous vehicle to an origination location of the plurality of origination locations;
selecting the autonomous vehicle to navigate to the origination location based at least in part on the routing data and the policy data;
generating command data for the autonomous vehicle using the routing data; and
sending the command data to the autonomous vehicle, the command data causing the autonomous vehicle to navigate to the origination location via the route.

US Pat. No. 9,517,767

INTERNAL SAFETY SYSTEMS FOR ROBOTIC VEHICLES

Zoox, Inc., Menlo Park, ...

1. A method for robotic vehicle internal safety system activation, comprising:
detecting that a seat of an autonomous vehicle is occupied;
detecting that a seat belt coupled with the seat is fastened;
applying, by a belt tensioner coupled with the seat belt, a first tension to seat belt;
autonomously navigating the autonomous vehicle toward a destination;
identifying an occurrence of a safety event from a plurality of safety related events, wherein the plurality of safety related
events include a breaking event, and an acceleration event;

increasing, by the belt tensioner, the first tension to a second tension based, at least in part, on the occurrence of the
safety event; and

reducing, by the belt tensioner, during the navigating, the second tension to the first tension.

US Pat. No. 10,304,191

THREE DIMENSIONAL BOUNDING BOX ESTIMATION FROM TWO DIMENSIONAL IMAGES

Zoox, Inc., Foster City,...

1. A system for estimating a three dimensional bounding box from a two dimensional image, the system including a non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the system to:receive an image captured from an image capture device;
detect an object in the image;
create a two dimensional bounding box around the object;
crop the image using the two dimensional bounding box to form a cropped image;
input the cropped image into a convolutional neural network; and
receive, based at least in part on the cropped image, from the convolutional neural network:
an orientation of a three dimensional bounding box associated with the object, wherein the orientation is indicative of a direction of motion associated with the object; and
dimensions of the three dimensional bounding box, wherein the dimensions comprise a length, a width, and a height;
wherein the convolutional neural network comprises:
a bin confidence branch configured to indicate a confidence value representing a coarse orientation of the three dimensional bounding box; and
an orientation offset branch configured to indicate an orientation offset with respect to the coarse orientation output by the bin confidence branch.

US Pat. No. 9,804,599

ACTIVE LIGHTING CONTROL FOR COMMUNICATING A STATE OF AN AUTONOMOUS VEHICLE TO ENTITIES IN A SURROUNDING ENVIRONMENT

Zoox, Inc., Menlo Park, ...

1. A method comprising:
accessing sensor data generated by a sensor system of an autonomous vehicle;
determining, based at least in part on the sensor data, a location of the autonomous vehicle within the environment, wherein
the location of the autonomous vehicle identifies a position and orientation of the autonomous vehicle within the environment
according to a global coordinate system;

calculating, based at least in part on the location of the autonomous vehicle and at least a portion of the sensor data, a
trajectory of the autonomous vehicle, wherein the trajectory indicates a planned path associated with navigating the autonomous
vehicle between at least a first location and a second location within the environment;

identifying, based at least in part on the sensor data, a pedestrian object within the environment;
determining a location of the pedestrian object in the environment, wherein the location of the pedestrian object identifies
a position and orientation of the pedestrian object within the environment according to the global coordinate system, wherein
the position identifies one or more coordinates and wherein the orientation identifies one or more of a pitch, a yaw, or a
roll of the pedestrian object;

determining, based at least in part on the location of the pedestrian object and the location of the autonomous vehicle, to
provide a visual alert in a direction of the pedestrian object;

selecting a light pattern from a plurality of light patterns, wherein a first one of light patterns is associated with a first
level of urgency of the visual alert to provide in the direction of the pedestrian object, and a second one of the light patterns
is associated with a second level of urgency of the visual alert to provide in the direction of the pedestrian object;

selecting, from a plurality of light emitters of the autonomous vehicle, a light emitter to provide the visual alert, wherein
the plurality of light emitters include a first light emitter located within a first area between a first wheel of the autonomous
vehicle and a second wheel of the autonomous vehicle, a second light emitter located within a second area between a third
wheel of the autonomous vehicle and a fourth wheel of the autonomous vehicle, a third light emitter located within a third
area between the first wheel and the third wheel, and a fourth light emitter located within a fourth area between the second
wheel and the fourth wheel; and

causing the light emitter to provide the visual alert in the direction to the pedestrian object, the light emitter emitting
light indicative of the light pattern into the environment.

US Pat. No. 10,395,444

VEHICLE SELF-DIAGNOSTICS

Zoox, Inc., Foster City,...

1. A system onboard a vehicle comprising:one or more processors; and
one or more-computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive sensor data from a plurality of sensors associated with the vehicle;
determine, based at least in part on analyzing at least a portion of the sensor data utilizing a model, a fault associated with the vehicle;
send a query to at least one component system associated with a component of the vehicle;
receive a response from the at least one component system;
determine, based at least in part on the response, that the fault is associated with the component;
determine, based at least in part on the fault associated with the component, at least one service issue associated with the vehicle; and
execute, by a control system of the vehicle, a command to redress the at least one service issue associated with the vehicle while the vehicle is driving, the command comprising altering operation of one or more components of the vehicle.

US Pat. No. 10,000,124

INDEPENDENT STEERING, POWER, TORQUE CONTROL AND TRANSFER IN VEHICLES

Zoox, Inc., Menlo Park, ...

1. A method performed by one or more processors, the method comprising:computing a trajectory for an autonomous vehicle;
causing the autonomous vehicle to navigate along the trajectory;
determining a nonoperational state of a first powertrain unit of the autonomous vehicle;
determining an operational state of a second powertrain unit;
disabling a propulsion operation of the second powertrain unit based at least in part on the nonoperational state of the first powertrain unit;
enabling a torque steering operation of the second powertrain unit; and
enabling a propulsion operation of a third powertrain unit and a propulsion operation of a fourth powertrain unit,
wherein the torque steering operation creates a yaw such that the autonomous vehicle continues to navigate along the trajectory.

US Pat. No. 10,137,856

SEATBELT SYSTEM INCLUDING OCCUPANT DETECTOR

Zoox, Inc., Foster City,...

1. A seatbelt system comprising:a seatbelt comprising:
a belt portion comprising a lap portion configured to constrain a waist of an occupant and a sash portion configured to constrain a torso of the occupant; and
a coupling configured to couple a portion of a vehicle to at least one of the lap portion or the sash portion;
a reel configured to receive one of at least a portion of the sash portion or at least a portion of the lap portion;
a tension sensor associated with at least one of the coupling, the belt portion, or the reel and configured to generate a signal indicative of tension in at least one of the lap portion or the sash portion;
a deployment sensor associated with one of the lap portion or the sash portion, the deployment sensor being configured to generate a signal indicative of a length of the lap portion or the sash portion deployed from the reel; and
an occupant detector in communication with the tension sensor and the deployment sensor and configured to determine, based at least in part on the signal indicative of tension and the signal indicative of the length, whether an occupant is properly wearing the seatbelt, such that the lap portion and the sash portion constrain the occupant.

US Pat. No. 9,606,539

AUTONOMOUS VEHICLE FLEET SERVICE AND SYSTEM

Zoox, Inc., Menlo Park, ...

1. A method comprising:
monitoring, at a computing system, a fleet of independent driverless vehicles, at least one of which is configured to autonomously
transit from a first geographic region to a second geographic region via a road network, the driverless vehicles being bidirectional
autonomous vehicles capable of driving forward in a first direction or driving forward in an opposite second direction and
having at least one mechanism for communicating a current forward direction to one or more potential people in a surrounding
environment;

receiving, at the computing system, data from a driverless vehicle in the fleet indicating an event encountered by the driverless
vehicle that may cause the driverless vehicle to maneuver from a current trajectory;

generating, by the computing system, multiple candidate trajectories for the driverless vehicle to maneuver from the current
trajectory;

calculating, by the computing system, confidence levels associated with the multiple candidate trajectories;
ranking, by the computing system, the multiple candidate trajectories according to the confidence levels;
choosing a particular candidate trajectory from among the multiple candidate trajectories based at least in part on the confidence
levels associated with the multiple candidate trajectories; and

transmitting, to the driverless vehicle, the particular candidate trajectory for use in maneuvering the driverless vehicle
whereby the driverless vehicle uses the mechanism to communicate any change of the first direction to the potential people
in the surrounding environment,

wherein the choosing comprises:
presenting, to a human teleoperator, at least a subset of the multiple candidate trajectories in ranked order according to
the associated confidence levels; and

receiving input from the human teleoperator to select the particular candidate trajectory.

US Pat. No. 10,242,457

AUGMENTED REALITY PASSENGER EXPERIENCE

Zoox, Inc., Foster City,...

1. An autonomous vehicle, comprising:a projector disposed in substantially a center of an interior portion of the autonomous vehicle;
one or more processors; and
one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining image data representing a passenger within the autonomous vehicle;
analyzing the image data to determine a pose of a head of the passenger;
determining an eye position relative to the head of the passenger;
determining a gaze direction of the passenger;
determining a surface within the autonomous vehicle for providing content based, at least in part, on the gaze direction and the eye position; and
causing the projector to output the content for the passenger at the surface.

US Pat. No. 10,053,088

OCCUPANT AWARE BRAKING SYSTEM

Zoox, Inc., Foster City,...

1. A vehicle comprising:one or more internal sensors to provide data associated with an interior portion of the vehicle; and
a cargo classification system in communication with at least the one or more internal sensors system to:
determine whether cargo is present in the interior portion of the vehicle based at least in part on one or more inputs from the one or more internal sensors; and
provide, to a vehicle control system, a classification of the cargo when present; and
the vehicle control system, in communication with the cargo classification system, the vehicle control system configured to maneuver the vehicle at or below a maximum acceleration based at least in part on data provided by the cargo classification system.

US Pat. No. 9,802,661

QUADRANT CONFIGURATION OF ROBOTIC VEHICLES

Zoox, Inc., Menlo Park, ...

1. An apparatus, comprising:
a first structural section including a first propulsion unit, a first steering unit and a first sensor unit;
a second structural section including a second propulsion unit, a second steering unit and a second sensor unit; and
one or more autonomous vehicle controllers electronically coupled with one or more of the first propulsion unit, the second
propulsion unit, the first steering unit, the second steering unit, the first sensor unit, or the second sensor unit.

US Pat. No. 10,303,961

OBJECT DETECTION AND PASSENGER NOTIFICATION

Zoox, Inc., Foster City,...

1. A vehicle comprising:one or more internal sensors to:
generate data associated with an interior portion of the vehicle; and
send the data to an object classification system;
the object classification system in communication with at least the one or more internal sensors to:
receive the data from the at least one of the one or more internal sensors;
input the data into a first machine learned model configured to determine a presence of an object in the vehicle;
input, based at least in part on the determination, a portion of the data associated with the object into a second machine learned model configured to output an object classification;
receive, from the second machine learned model, the object classification;
memory storing instructions that, when executed by one or more processors, cause the one or more processors to:
determine that the vehicle is within a distance from a destination; and
send an indication to a notification system that the vehicle is within the distance from the destination; and
the notification system to:
receive the indication from the navigation system; and
provide one or more of an audio or a visual reminder to a user to remove the object from the vehicle based at least on the object classification and the indication.

US Pat. No. 10,134,182

LARGE SCALE DENSE MAPPING

Zoox, Inc., Foster City,...

1. A computer-implemented method, comprising:determining an array of cells to represent, in a signed distance function (SDF) volumetric representation, at least a portion of an environment;
obtaining sensor data representative of the environment from one or more sensors on a system, a sensor of the one or more sensors including a sensor center;
determining a ray originating at the sensor center and passing through a point associated with the sensor data;
determining a segment along the ray within a distance, d, of the point, the segment having a length that is less than a portion of the ray traversing the SDF volumetric representation;
identifying one or more cells, of the array, intersected by the segment, each cell of the one or more cells comprising a plurality of voxels;
determining a hash entry for at least a portion of the one or more cells;
allocating memory for each of the one or more cells; and
updating, within the memory and using at least a portion of the sensor data, cell data associated with the one or more cells to update the SDF volumetric representation.

US Pat. No. 10,303,174

INTERNAL SAFETY SYSTEMS FOR ROBOTIC VEHICLES

Zoox, Inc., Foster City,...

1. A method for vehicle safety system activation, comprising:receiving sensor data from a sensor associated with a vehicle;
determining a vehicle location within an environment, wherein the vehicle location identifies one or more of a position or an orientation of the vehicle within the environment;
calculating, based at least in part on the vehicle location, a vehicle trajectory, wherein the vehicle trajectory indicates a subset of a planned path associated with navigating the vehicle between at least a first location and a second location within the environment;
identifying an object within the environment;
determining an object location, wherein the object location identifies one or more of a position or an orientation of the object within the environment;
determining an object trajectory based at least in part on the object location; and
causing a safety system to activate based at least in part on the vehicle trajectory and the object trajectory.

US Pat. No. 10,118,590

SEATBELT SYSTEM INCLUDING OCCUPANT DETECTOR

Zoox, Inc., Foster City,...

1. A seatbelt system comprising:a seatbelt comprising:
a belt portion comprising a lap portion configured to constrain a waist of an occupant and a sash portion configured to constrain a torso of the occupant; and
a coupling configured to couple a portion of a vehicle to at least one of the lap portion or the sash portion;
a reel configured to receive one of at least a portion of the sash portion or at least a portion of the lap portion;
a tension sensor associated with at least one of the coupling, the belt portion, or the reel and configured to generate a signal indicative of tension in at least one of the lap portion or the sash portion;
a deployment sensor associated with one of the lap portion or the sash portion, the deployment sensor being configured to generate a signal indicative of a length of the lap portion or the sash portion deployed from the reel; and
an occupant detector in communication with the tension sensor and the deployment sensor and configured to determine, based at least in part on the signal indicative of tension and the signal indicative of the length, whether an occupant is properly wearing the seatbelt, such that the lap portion and the sash portion constrain the occupant.

US Pat. No. 9,720,415

SENSOR-BASED OBJECT-DETECTION OPTIMIZATION FOR AUTONOMOUS VEHICLES

Zoox, Inc., Menlo Park, ...

1. A method comprising:
identifying a sensor anomaly at an autonomous vehicle system, the autonomous vehicle system comprising an autonomous vehicle
having a plurality of sensors;

determining a type of the sensor anomaly;
retrieving, from a data store and based on the sensor anomaly, data representing at least one sensor recovery strategy, the
at least one sensor recovery strategy including one or more courses of action to modify operation of the autonomous vehicle
in response to the sensor anomaly;

selecting a course of action from the one or more courses of action based at least in part on the type of the sensor anomaly;
and

executing the course of action at the autonomous vehicle system, wherein executing the course of action causes the autonomous
vehicle to operate within a range of operational parameters defined by the course of action, wherein

the range of operational parameters comprises a particular trajectory, and
executing the course of action causes the autonomous vehicle system to perform a maneuver to orient the autonomous vehicle
such that the autonomous vehicle is substantially aligned with the particular trajectory and the particular trajectory is
within respective fields of view of a majority of the plurality of sensors.

US Pat. No. 9,916,703

CALIBRATION FOR AUTONOMOUS VEHICLE OPERATION

Zoox, Inc., Menlo Park, ...

1. A method comprising:
receiving data at an autonomous vehicle system comprising a plurality of sensors, the plurality of sensors including an inertial
measurement unit (IMU) sensor, an odometry sensor, and a multi-beam LIDAR sensor, and the data representing measurements made
using the plurality of sensors;

determining a distance traveled by the autonomous vehicle system based on one or more sensor measurements of the odometry
sensor;

determining a velocity of the autonomous vehicle system based on one or more sensor measurements of the IMU sensor;
identifying an abnormal sensor measurement, made using the multi-beam LIDAR sensor, based at least in part on the data;
generating an initial calibration parameter, associated with the multi-beam LIDAR sensor, based at least in part on the abnormal
sensor measurement;

generating an expected sensor measurement, associated with the multi-beam LIDAR sensor, based on the data;
modifying the initial calibration parameter based at least in part on the expected sensor measurement, wherein modifying the
initial calibration parameter includes generating a modified calibration parameter associated with the multi-beam LIDAR sensor;
and

calibrating the multi-beam LIDAR sensor based at least in part on the modified calibration parameter, the distance traveled
by the autonomous vehicle system, and the velocity of the autonomous vehicle system.

US Pat. No. 10,176,388

SPATIAL AND TEMPORAL INFORMATION FOR SEMANTIC SEGMENTATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive a plurality of images from an image capture device;
select a subset of the plurality of images, the subset of the plurality of images comprising at least a first image associated with a first time and a second image associated with a second time after the first time, wherein the subset of the plurality of images comprises fewer images than the plurality of images;
transmit the subset of the plurality of images into a convolutional neural network; and
generate a plurality of output images comprising at least one predictive segmented image, wherein the at least one predictive segmented image of the plurality of output images is associated with a third time after the second time and comprises predictive segmentation information,
wherein the convolutional neural network comprises:
one or more encoder layers configured to receive, as a plurality of input images, the subset of the plurality of images and configured to output an encoded output,
one or more long short term memory (LSTM) layers, the one or more LSTM layers configured to receive the encoded output and configured to output an LSTM output, and
one or more decoder layers configured to receive the LSTM output and configured to output the plurality of output images.

US Pat. No. 9,612,123

ADAPTIVE MAPPING TO NAVIGATE AUTONOMOUS VEHICLES RESPONSIVE TO PHYSICAL ENVIRONMENT CHANGES

Zoox, Inc., Menlo Park, ...

1. A method comprising:
receiving, at a computing system, a first type of sensor data from multiple driverless vehicles in a fleet of autonomous driverless
vehicles, wherein the first type of sensor data is acquired by a first type of sensors on the driverless vehicles to sense
objects in environments encountered while the driverless vehicles are driving along roads;

receiving, at the computing system, a second type of sensor data from the multiple driverless vehicles in the fleet of autonomous
driverless vehicles, wherein the second type of sensor data is acquired by a second type of sensors on the driverless vehicles
to sense objects in the environments encountered while the driverless vehicles are driving along the roads;

storing, at the computing system, the first type of sensor data and the second type of sensor data;
accessing, by the computing system and for a particular driverless vehicle, subsets of the first type of sensor data and subsets
of the second type of sensor data;

aligning, by the computing system and for the particular driverless vehicle, the subsets of the first type of sensor data
with the subsets of the second type of sensor data to provide aligned sensor data pertaining to objects in an environment
encountered by the particular driverless vehicle;

aligning, by the computing system and for the particular driverless vehicle, the aligned sensor data relative to positioning
map data pertaining to a global coordinate system to localize the aligned sensor data to a location of the particular driverless
vehicle;

generating, by the computing system, data sets of three-dimensional map data based on the aligned sensor data and the positioning
map data;

detecting a change between the generated data sets of three-dimensional map data and a stored data set of three-dimensional
map data, the change being representative of a state change in the environment surrounding the particular driverless vehicle;

updating, at the computing system, the stored data set of three-dimensional map data to generate an updated data set of three-dimensional
map data reflecting the state change in the environment; and

transmitting the updated data set of three-dimensional map data to the multiple driverless vehicles in the fleet of autonomous
driverless vehicles,

wherein at least one of the driverless vehicles is controlled based at least in part on the updated data set of three-dimensional
map data.

US Pat. No. 10,289,984

AUTOMATED VEHICLE DIAGNOSTICS AND MAINTENANCE

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more non-transitory computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive at least one of sensor data or one or more error codes associated with an autonomous vehicle;
determine at least one servicing issue associated with the autonomous vehicle based at least in part on the at least one of the sensor data or the one or more error codes, the at least one servicing issue associated with a cleanliness of the autonomous vehicle, a potential hardware malfunction associated with the autonomous vehicle, or a state of the autonomous vehicle;
determine that the autonomous vehicle is to be serviced by a mobile technician based at least in part on the at least one servicing issue associated with the autonomous vehicle;
determine a technician skill level associated with the at least one servicing issue, the technician skill level indicating a qualification of technicians to resolve the at least one servicing issue;
select a technician from one or more technicians to service the at least one servicing issue associated with the autonomous vehicle, wherein the technician is selected based at least in part on i) the technician skill level, ii) a distance between a first location of the autonomous vehicle and a second location of the technician, and iii) a traffic level associated with the first location or the second location; and
provide instructions to the autonomous vehicle to navigate to the second location of the technician.

US Pat. No. 10,001,775

MACHINE LEARNING SYSTEMS AND TECHNIQUES TO OPTIMIZE TELEOPERATION AND/OR PLANNER DECISIONS

Zoox, Inc., Foster City,...

1. An autonomous vehicle comprising:one or more processors;
a communication interface, the communication interface configured to establish a communications link between the autonomous vehicle and one or more other devices or autonomous vehicles;
one or more sensors; and
memory having stored thereon processor-executable instructions that, when executed by the one or more processors, configure the autonomous vehicle to perform operations comprising:
obtaining sensor data from the one or more sensors;
detecting, from the sensor data, an event, the event being associated with operation of the autonomous vehicle;
determining a type of the event;
requesting, via the communication interface, a teleoperator command based, at least in part, on the type of the event;
receiving via the communication interface a teleoperator command; and
controlling the autonomous vehicle based, at least in part, on the teleoperator command.

US Pat. No. 9,878,689

SEATBELT SYSTEM INCLUDING OCCUPANT DETECTOR

Zoox, Inc., Menlo Park, ...

1. A seatbelt system comprising:
a seatbelt comprising:
a belt portion comprising a lap portion configured to constrain a waist of an occupant and a sash portion configured to constrain
a torso of the occupant; and

a coupling configured to couple a portion of a vehicle to at least one of the lap portion or the sash portion;
a reel configured to receive one of at least a portion of the sash portion or at least a portion of the lap portion;
a tension sensor associated with at least one of the coupling, the belt portion, or the reel and configured to generate a
signal indicative of tension in at least one of the lap portion or the sash portion;

a deployment sensor associated with one of the lap portion or the sash portion, the deployment sensor being configured to
generate a signal indicative of a length of the lap portion or the sash portion deployed from the reel;

a transmitter associated with the sash portion and configured to a transmit signal indicative of a location of the transmitter;
a signal receiver configured to couple to a portion of the vehicle and receive the signal transmitted by the transmitter;
and

an occupant detector in communication with the tension sensor and the deployment sensor and configured to determine, based
at least in part on the signal indicative of tension and the signal indicative of the length, whether an occupant is properly
wearing the seatbelt, such that the lap portion and the sash portion constrain the occupant,

wherein the occupant detector is configured to receive the signal from the signal receiver and determine whether the occupant
is properly wearing the seatbelt based in part on the signal received from the signal receiver.

US Pat. No. 9,632,502

MACHINE-LEARNING SYSTEMS AND TECHNIQUES TO OPTIMIZE TELEOPERATION AND/OR PLANNER DECISIONS

Zoox, Inc., Menlo Park, ...

1. A method, comprising:
receiving, from one or more of a sensor or a communication interface, telemetry data associated with an event, the event being
a situation or condition associated with operation of an autonomous vehicle;

obtaining policy data, the policy data including instructions for operating an autonomous vehicle according to a trajectory
and the policy data including a confidence level associated with the received trajectory, the confidence level indicating
a degree of certainty that the autonomous vehicle, in operating according to the trajectory, will operate safely in view of
the event;

obtaining candidate trajectories responsive to the event, based on the telemetry data, each candidate trajectory having an
associated confidence level;

generating, by machine-learning by a processor and based at least in part on the candidate trajectories and the telemetry
data, updated policy data that includes instructions for operating the autonomous vehicle responsive to the event differently
than according to the policy data; and

communicating the updated policy data to at least one autonomous vehicle via a communications interface.

US Pat. No. 10,332,292

VISION AUGMENTATION FOR SUPPLEMENTING A PERSON'S VIEW

Zoox, Inc., Foster City,...

1. An augmentation system for supplementing a person's view from a vehicle of an environment through which the vehicle travels, the augmentation system comprising:a sensor on the vehicle configured to generate sensor data, the sensor data representative of an object in the environment through which the vehicle travels;
a display; and
one or more processors configured to:
determine a direction of view of the person;
determine, based at least in part on the direction of view, a subset of the sensor data;
determine, based at least in part on the subset of the sensor data, a representation of the object in the environment through which the vehicle travels; and
cause the display to augment the person's view of the environment through which the vehicle travels by presenting the subset of the sensor data superimposed with the representation of the object,
wherein the representation of the object is indicative of at least one of a location of the object, one or more extents of the object, a type of the object, a historical path of the object, or a predicted path of the object.

US Pat. No. 10,384,718

VEHICLE PARKING ASSIST

Zoox, Inc., Foster City,...

1. A system associated with an autonomous vehicle for providing parking assistance to a parking vehicle, the system comprising:one or more sensors on the autonomous vehicle;
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform actions comprising:
receiving sensor data from the one or more sensors, wherein at least a portion of the sensor data is associated with the parking vehicle;
determining, based at least in part on the sensor data, the parking vehicle is moving toward a parking location proximate the autonomous vehicle;
tracking, based at least in part on the sensor data, a position, an orientation, or a velocity of the parking vehicle with respect to the parking location; and
causing, based at least in part on tracking the position, the orientation, or the velocity, an indication to be output via one or more visual or audio outputs associated with the autonomous vehicle, the indication providing one or more instructions for guiding the parking vehicle into the parking location, wherein the indication comprises at least one of:
a light beam output via a light emitter associated with the autonomous vehicle, the light beam forming a visual indication associated with the one or more instructions; or
an acoustic beam that is output via a speaker associated with the autonomous vehicle.

US Pat. No. 10,248,119

INTERACTIVE AUTONOMOUS VEHICLE COMMAND CONTROLLER

Zoox, Inc., Foster City,...

1. A method comprising:receiving, from a sensor disposed in an autonomous vehicle, a signal corresponding to an audible request to control a route of the autonomous vehicle;
extracting one or more words from the signal;
determining one or more generative probabilistic models for interpreting the one or more words;
determining, based at least in part on the one or more generative probabilistic models, a probability associated with a first word of the one or more words;
determining that the probability is equal to or below a threshold value;
sending, based at least in part on the probability being equal to or below the threshold value, the signal to a teleoperator system;
receiving, from the teleoperator system, a command to control the route of the autonomous vehicle system in accordance with the request;
determining a plurality of courses of action based on the command;
for each of the plurality of courses of action, determining at least one second probabilistic model;
based on the second probabilistic model, determining a confidence level for each of the plurality of courses of action;
forming a subset of the plurality of courses of action, each course of action in the subset having a confidence level of at least a threshold confidence level; and
executing a course of action from the subset at the autonomous vehicle system.

US Pat. No. 9,910,441

ADAPTIVE AUTONOMOUS VEHICLE PLANNER LOGIC

Zoox, Inc., Menlo Park, ...

1. A method comprising:
receiving path data with which to guide motion of an autonomous vehicle from a first geographic location to a second geographic
location;

receiving perception data from a perception engine, the perception data comprising static map data, current object state data,
and predicted object state data;

receiving, from a localizer, local pose data of the autonomous vehicle;
substantially simultaneously generating at least a plurality of trajectories and a plurality of contingent trajectories based,
at least in part, on the path data, the local pose data, the static map data, the current object state data, and the predicted
object state data;

selecting a trajectory from the plurality of trajectories to use to control one or more vehicle components;
selecting a contingent trajectory from the plurality of contingent trajectories to use to control the one or more vehicle
components;

controlling the one or more vehicle components of the autonomous vehicle according to the trajectory to direct motion of the
autonomous vehicle along the trajectory;

monitoring generation of the trajectory;
detecting cessation of the generation of the trajectory; and
responsive to detecting the cessation of the generation of the trajectory, controlling the one or more vehicle components
according to the contingent trajectory.

US Pat. No. 10,275,662

ESTIMATING FRICTION BASED ON IMAGE DATA

Zoox, Inc., Foster City,...

1. A friction estimation system for estimating friction-related data associated with a surface on which a vehicle travels, the friction estimation system comprising:a camera array comprising a plurality of imagers configured to capture image data associated with a surface on which a vehicle travels;
an image interpreter in communication with the camera array and configured to receive the image data from the camera array and determine friction-related data associated with the surface based, at least in part, on the image data, the image interpreter being configured to be in communication with a vehicle control system and provide the friction-related data to the vehicle control system;
a location sensor, an output of which comprises a location of the vehicle; and
one or more processors configured to associate, based at least in part on the location, the friction-related data with a map accessible to the vehicle.

US Pat. No. 10,189,476

BRAKE FORCE DISTRIBUTION

Zoox, Inc., Foster City,...

1. A vehicle operable to travel in at least a first direction, the vehicle comprising:two or more wheel assemblies;
a power system operable to provide power to enable the vehicle to accelerate;
a brake system operable to cause, responsive to a braking action, a first negative force to be distributed between the two or more wheel assemblies to cause the vehicle to decelerate; and
a control system comprising:
one or more processors; and
one or more instructions executable by the one or more processors to send a command, responsive to the braking action, to at least one of the power system or the brake system to:
cause the power system to affect a second negative force on a first wheel assembly of the two or more wheel assemblies at a first time; and
cause the power system to affect a positive force on a second wheel assembly of the two or more wheel assemblies at a second time that is substantially a same time as the first time,
wherein an application of the second negative force and the positive force generates a brake force distribution such that a first total amount of braking force applied at the first wheel assembly is greater than a second total amount of braking force applied at the second wheel assembly.

US Pat. No. 9,701,239

SYSTEM OF CONFIGURING ACTIVE LIGHTING TO INDICATE DIRECTIONALITY OF AN AUTONOMOUS VEHICLE

Zoox, Inc., Menlo Park, ...

1. A method for visual indication of direction of travel of a bidirectional autonomous vehicle, comprising:
receiving data representing a trajectory of the bidirectional autonomous vehicle in an environment;
causing the bidirectional autonomous vehicle to navigate in a first direction of travel that corresponds to the trajectory,
wherein a first end of the bidirectional autonomous vehicle is facing the first direction of travel;

selecting data representing a first directional light pattern, wherein the first directional light pattern is configured to
indicate the first direction of travel of the bidirectional autonomous vehicle;

selecting a light emitter of the bidirectional autonomous vehicle to emit light into the environment;
causing, using the data representing the first directional light pattern, the light emitter to emit the light to visually
convey the first direction of travel of the bidirectional autonomous vehicle;

detecting that the bidirectional autonomous vehicle is switching from navigating in the first direction of travel to a second
direction of travel, wherein a second end of the bidirectional autonomous vehicle is facing the second direction of travel;

selecting data representing a second directional light pattern, wherein the second directional light pattern is configured
to indicate the second direction of travel of the bidirectional autonomous vehicle; and

causing, using the data representing the second directional light pattern, the light emitter to emit the light to visually
convey the second direction of travel of the bidirectional autonomous vehicle,

wherein the second directional light pattern and the first directional light pattern are substantially symmetric about an
axis of the bidirectional autonomous vehicle.

US Pat. No. 10,322,724

BRAKE FORCE DISTRIBUTION

Zoox, Inc., Foster City,...

1. A computer-implemented method comprising:causing, by a brake system of a vehicle, a first force to be applied to a first wheel of the vehicle and a second force to be applied to a second wheel of the vehicle, wherein the first force and the second force are applied in a first direction that is opposite a second direction in which the vehicle is travelling; and
sending, by a control system of the vehicle, a command instructing a power system of the vehicle to affect a third force on the first wheel, wherein the third force is applied in the second direction.

US Pat. No. 10,317,240

TRAVEL DATA COLLECTION AND PUBLICATION

Zoox, Inc., Foster City,...

1. A computer implemented method comprising:receiving data from one or more sensors, the data associated with at least one of an event or an area of interest;
determining, from a plurality of classifications and based at least in part on a semantic scene parsing, a classification for the at least one of the event or the area of interest, wherein the plurality of classifications comprises:
a first classification associated with an accident;
a second classification associated with a construction site; and
a third classification associated with a fire;
determining, based at least in part on the classification, that the data is relevant data;
incorporating, at a global map server, the relevant data and the classification into a global map;
receiving a request for derived data;
determining, based at least in part on the global map, the derived data; and
causing, based at least in part on the derived data, an autonomous vehicle to be deployed to a location associated with the at least one of the event or the area of interest in order to gather additional data associated with the at least one of the event or the area of interest.

US Pat. No. 10,589,679

PROVIDING VISUAL REFERENCES TO PREVENT MOTION SICKNESS IN VEHICLES

Zoox, Inc., Foster City,...

1. A system comprising:one or more video sources disposed in an interior of a vehicle;
one or more sensors to provide data related to motion of the vehicle; and
a controller, comprising one or more processors, in communication with at least the one or more projectors and the one or more sensors; and
memory storing computer-executable instructions that, when executed by the one or more processors, cause the system to:
receive data from the one or more sensors regarding the motion of the vehicle;
cause the one or more video sources to display one or more visual references on a surface of the interior of the vehicle based at least in part on data from the one or more sensors, the one or more visual references including a portion of an object outside the vehicle that, relative to a point inside a passenger compartment of the vehicle, is obscured by the surface of the interior of the vehicle, the one or more visual references being adjusted on the one or more video sources based at least in part on tracked gazes associated with a number of passengers inside of the vehicle using vision-based eye-tracking, wherein the tracked (lazes do not intersect;
detect, based at least in part on the data regarding the motion of the vehicle, a rotational motion of the vehicle in a first direction; and
based at least in part on the rotational motion of the vehicle, cause the one or more video sources to rotate the one or more visual references as displayed by the one or more video sources relative to the point inside the passenger compartment of the vehicle as the vehicle rotates such that the object appears unobscured relative to the point inside the passenger compartment of the vehicle, wherein the one or more visual references are rotated by the one or more video sources in a second direction opposite to the first direction.

US Pat. No. 10,259,514

DRIVE MODULE FOR ROBOTIC VEHICLE

Zoox, Inc., Foster City,...

1. A drive module, comprising:at least one propulsion unit, the at least one propulsion unit including a motor to drive at least one wheel;
at least one steering unit, the at least one steering unit configured to steer the at least one wheel;
at least one sensor unit configured to sense an environment external to the drive module;
at least one power unit, the at least one power unit being in electrical communication with the at least one propulsion unit, the at least one steering unit, and the at least one sensor unit;
a controller in electrical communication with the at least one power unit and configured to control the at least one propulsion unit and the at least one steering unit based at least in part on input from the at least one sensor unit; and
one or more couplings configured to couple the drive module to a structural section of a component of a vehicle.

US Pat. No. 9,878,664

METHOD FOR ROBOTIC VEHICLE COMMUNICATION WITH AN EXTERNAL ENVIRONMENT VIA ACOUSTIC BEAM FORMING

Zoox, Inc., Menlo Park, ...

1. A method comprising:
calculating a trajectory of an autonomous vehicle in an environment, the autonomous vehicle including an acoustic beam steering
array, the acoustic beam steering array including at least a first speaker adjacent to a second speaker, wherein the first
speaker is spaced from the second speaker by a wave front distance;

determining a location of an object disposed in the environment, the object having an object type;
predicting an object trajectory of the object based at least in part on the object type and the location of the object;
estimating a threshold location in the environment based at least in part on the object trajectory and the trajectory of the
autonomous vehicle, the threshold location associated with an acoustic alert;

selecting an audio signal to associate with the acoustic alert;
detecting that the location of the object is coincident with the threshold location;
determining, based at least in part on the location of the object, a direction of propagation to emit a beam of steered acoustic
energy;

calculating, based at least in part on the wave front distance and a speed of sound, a first signal delay associated with
the first speaker;

calculating, based at least in part on the wave front distance and the speed of sound, a second signal delay associated with
the second speaker; and

causing the acoustic beam steering array to emit the beam of steered acoustic energy by at least delaying-application of the
audio signal to a signal input of one or more amplifiers coupled with the first speaker and the second speaker by one or more
times based at least on one or more of the first signal delay or the second signal delay.

US Pat. No. 10,509,410

EXTERNAL CONTROL OF AN AUTONOMOUS VEHICLE

Zoox, Inc., Foster City,...

1. A vehicle comprising:one or more sensors configured to sense objects in an environment of the vehicle;
one or more drive systems configured to cause the vehicle to move in the environment;
a control system configured to:
receive, from a source external to the vehicle, one or more control commands comprising a command for the vehicle to proceed in a direction at a commanded speed equal to or below about 10 km/h;
receive sensor data from the one or more sensors, the sensor data comprising information about an object in the environment;
determine a safety region proximate the vehicle;
determine, from the sensor data, a distance of the object from the vehicle;
determine, based at least in part on the distance, that the object is in the safety region;
determine, based at least in part on the object being in the safety region and the distance, a modified speed for the vehicle less than the commanded speed; and
control the one or more drive systems to move the autonomous vehicle in the direction at the modified speed.

US Pat. No. 10,543,838

ROBOTIC VEHICLE ACTIVE SAFETY SYSTEMS AND METHODS

Zoox, Inc., Foster City,...

1. A method comprising:receiving trajectory data representing a trajectory of an autonomous vehicle in an environment external to the autonomous vehicle;
determining object data comprising a position of an object in the environment, an object type of the object, and an object classification of the object;
determining a predicted location of the object in the environment based at least in part on the object data;
determining that the predicted location of the object coincides with the trajectory;
determining a first threshold location along the trajectory based at least in part on the predicted location of the object and the object data;
determining a second threshold location along the trajectory based at least in part on the predicted location of the object and the object data;
sensing an updated object position of the object in the environment;
implementing a safety action using a safety system based at least in part on the updated object position, the safety action associated with the first threshold location or the second threshold location.

US Pat. No. 10,589,738

ELECTRONIC BRAKING SYSTEMS AND METHODS

Zoox, Inc., Foster City,...

14. A system comprising:an electronic brake caliper assembly, mounted proximate a wheel of a vehicle, the electronic brake caliper assembly comprising:
a brake caliper;
one or more brake pads, mounted in the brake caliper, the one or more brake pads to generate friction on a brake rotor mechanically coupled to the wheel;
a brake actuator configured to apply a force to the one or more brake pads; and
a caliper electronic control unit (CECU), in communication with a wheel speed sensor and an executive motion unit (EMU), the CECU configured to:
receive a first signal to provide braking from the EMU;
send a second signal to the brake actuator of the electronic brake caliper assembly to apply a first force to a brake pad of the one or more brake pads on the electronic brake caliper assembly to create a first braking torque on the brake rotor mechanically coupled to the wheel;
receive a third signal from the wheel speed sensor located proximate the wheel indicative of a current wheel speed of the wheel;
compare the current wheel speed to an expected wheel speed; and
send a fourth signal to the brake actuator to apply a second force in response to the comparison of the current wheel speed to the expected wheel speed.

US Pat. No. 10,535,138

SENSOR DATA SEGMENTATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions executable by the one or more processors to:
receive a plurality of images representing an environment, the plurality of images comprising image data generated by one or more image capture devices;
transmit the image data to an image segmentation network configured to segment the plurality of images and generate segmented images;
receive first sensor data associated with the environment, the first sensor data received from a light detection and ranging (LIDAR) sensor;
associate the first sensor data with the segmented images to create a training dataset;
transmit the training dataset to a machine learning network configured to run a sensor data segmentation model; and
train the sensor data segmentation model using the training dataset such that, upon receiving input comprising additional LIDAR data and excluding additional image data, the sensor data segmentation model is configured to output segmented additional LIDAR sensor data.

US Pat. No. 10,446,037

SOFTWARE APPLICATION TO REQUEST AND CONTROL AN AUTONOMOUS VEHICLE SERVICE

Zoox, Inc., Foster City,...

1. A method comprising:receiving, from one or more sensors associated with a vehicle, sensor data indicative of an environment through which the vehicle is travelling;
receiving data representing a pose of the vehicle, the data representing the pose based at least in part on the sensor data from the one or more sensors associated with the vehicle;
receiving map data indicative of the environment;
causing display of an interactive map on a user interface, the interactive map comprising a representation of the map data and a rendering of the data representing the pose of the vehicle;
receiving, via a user interaction with the interactive map on the user interface, a request indicating a destination; and
controlling the vehicle to travel to the destination.

US Pat. No. 10,713,504

ESTIMATING FRICTION BASED ON IMAGE DATA

Zoox, Inc., Foster City,...

1. A system for estimating friction-related data associated with a surface on which a vehicle travels, the system comprising:a sensor configured to capture sensor data associated with the surface on which the vehicle travels; and
one or more processors configured to:
determine friction-related data associated with the surface based, at least in part, on the sensor data;
provide a correlation between the sensor data and the friction-related data, the correlation comprising one or more of a correlation between the sensor data and a material from which the surface is made, a correlation between the material and the friction-related data, a correlation table correlating the sensor data and the friction-related data, or a mathematical relationship between the sensor data and the friction-related data;
receive a location of the vehicle; and
associate, based at least in part on the location, the friction-related data with a map accessible to the vehicle.

US Pat. No. 10,372,130

COMMUNICATING REASONS FOR VEHICLE ACTIONS

Zoox, Inc., Foster City,...

1. An autonomous vehicle comprising:one or more processors;
one or more interior output devices disposed in an interior of the autonomous vehicle;
one or more sensors to provide data related to an environment of the autonomous vehicle; and
one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
causing the autonomous vehicle to traverse a path to a destination location by controlling the motion of the autonomous vehicle according to multiple trajectories calculated for traversing the path;
receiving sensor data from the one or more sensors;
detecting, based at least in part on the sensor data, an event;
determining an action to be taken by the autonomous vehicle responsive to the event;
determining a semantic meaning of the event and the action to be taken responsive to the event;
outputting, by the one or more interior output devices, an indication of the event and the action; and
causing the autonomous vehicle to perform the action.

US Pat. No. 10,334,050

SOFTWARE APPLICATION AND LOGIC TO MODIFY CONFIGURATION OF AN AUTONOMOUS VEHICLE

Zoox, Inc., Foster City,...

1. A method comprising:receiving an indication of a location of a user device;
generating instructions dispatching an autonomous vehicle system to the location to make the autonomous vehicle system available to a user to transport the user to a destination;
transmitting the instructions to the autonomous vehicle system; and
transmitting information associated with the user to the autonomous vehicle system, the information including road handling setting information including configuration data associated with the user, the configuration data including one or more settings associated with an active suspension system of the autonomous vehicle system, the configuration data being used by the autonomous vehicle to modify an active suspension parameter of the active suspension system, prior to the user entering the autonomous vehicle at the location.

US Pat. No. 10,471,923

OCCUPANT PROTECTION SYSTEM INCLUDING EXPANDABLE CURTAIN AND/OR EXPANDABLE BLADDER

Zoox, Inc., Foster City,...

1. An occupant protection system for a vehicle, the occupant protection system comprising:an expandable curtain configured to be expanded from a stowed state to a deployed state having a length configured to extend at least a portion of a distance between a vehicle roof and a vehicle floor, wherein the expandable curtain in the deployed state comprises:
a first side configured to extend along a portion of a first interior side of the vehicle;
a second side configured to extend along a portion of a second interior side of the vehicle opposite the first interior side; and
a transverse portion extending between the first side and the second side, the transverse portion being configured to divide the vehicle,
wherein the first side, the second side, and the transverse portion of the expandable curtain are contiguous;
an expandable bladder comprising an occupant facing surface and a rear surface opposite the occupant facing surface, the expandable bladder configured to expand from a stowed state to a deployed state, such that in the deployed state the transverse portion of the expandable curtain comes into contact with the rear surface of the expandable bladder to provide a reaction surface for the expandable bladder; and
a deployment control system configured to cause the expandable curtain to expand from the stowed state to the deployed state.

US Pat. No. 10,509,947

CONVERTING MULTI-DIMENSIONAL DATA FOR IMAGE ANALYSIS

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more non-transitory computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
capture a three-dimensional dataset using at least one LIDAR sensor installed on an autonomous vehicle;
determine a projection shape for mapping at least a portion of the three-dimensional dataset to the projection shape, the projection shape having at least a shape comprising one or more of a sphere or a cylinder being oriented based at least in part on a position of the at least one LIDAR sensor, the shape being discretized into a number of cells, the number of cells representing a resolution of the shape;
project, as projected data, the at least the portion of the three-dimensional dataset to one or more cells associated with the projection shape, wherein data associated with a data point of the at least the portion of the three-dimensional dataset is stored in association with a corresponding cell of the projection shape;
convert the projected data to a multi-channel two-dimensional image, wherein the multi-channel two-dimensional image comprises at least a first channel storing first information associated with a first coordinate of the data point and a second channel storing second information associated with a second coordinate of the data point; and
input the multi-channel two-dimensional image into at least one segmenter and at least one classifier to generate segmentation information and classification information.

US Pat. No. 10,499,180

THREE-DIMENSIONAL SOUND FOR PASSENGER NOTIFICATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable instructions that, when executed by the one or more processors, cause the system to:
receive sensor data from a sensor in an interior of an autonomous vehicle;
determine, based at least in part on the sensor data, a first location of a passenger in the autonomous vehicle;
determine, based on a vehicle system onboard the autonomous vehicle, an occurrence of an event associated with the passenger in the autonomous vehicle;
send, based at least in part on determining the occurrence of the event, a signal associated with a three-dimensional (3D) sound to a speaker system configured to direct sound to an interior of the autonomous vehicle, the 3D sound corresponding to the event; and
cause the 3D sound to be output by the speaker system based at least in part on the signal and the first location of the passenger, wherein the 3D sound, when output by the speaker system, is configured to be perceived as though the 3D sound is being emitted from a second location, and wherein the second location is different from, and is determined based on, the first location of the passenger in the autonomous vehicle.

US Pat. No. 10,496,766

SIMULATION SYSTEM AND METHODS FOR AUTONOMOUS VEHICLES

Zoox, Inc., Foster City,...

1. A method, comprising:receiving first data representing a dynamic object in a physical environment;
determining, by a vehicle controller, a classification of the dynamic object based at least in part on the first data;
determining second data representing dynamic-related characteristics associated with the dynamic object;
generating, by the vehicle controller, a data model of the dynamic object based at least in part on at least one of the first data, the second data, or the classification;
simulating, by the vehicle controller and based at least in part on the data model of the dynamic object, a predicted range of motion of the dynamic object in a simulated environment;
generating, by the vehicle controller and based at least in part on the predicted range of motion of the dynamic object, a command for controlling an autonomous vehicle; and
controlling, based at least in part on the command, the autonomous vehicle.

US Pat. No. 10,488,172

INDEPENDENT CONTROL OF VEHICLE WHEELS

Zoox, Inc., Foster City,...

1. A wheel control system for a vehicle having a longitudinal axis extending along a line between a first end of the vehicle and a second end of the vehicle opposite the first end of the vehicle, the wheel control system comprising:a first steering assembly associated with first wheels;
a wheel controller configured to:
receive a signal indicative of at least one of torque supplied to the first wheels of the vehicle or a lateral force associated with the first steering assembly;
control, based at least in part on the signal, steering angles and one or more of a toe angle, camber angle, or caster angle of the first wheels;
determine misalignment of one or more of the first wheels while the vehicle is travelling; and
at least one of reduce or eliminate the misalignment.

US Pat. No. 10,633,908

VEHICLE DOOR ACTUATOR ASSEMBLY

Zoox, Inc., Foster City,...

1. A vehicle comprising:a vehicle body defining an interior space;
a vehicle door with a door frame;
a mount connected to the vehicle body;
an arm coupled to the door frame at a first pivot point and coupled to the mount at a second pivot point, wherein the arm is a goose neck arm which comprises:
a curved portion having a first end and a second end;
a first elongated straight portion extending from the first end of the curved portion in a first direction; and
a second elongated straight portion extending from the second end of the curved portion in the first direction; and
an actuator coupled to the arm to cause the arm to pivot about the second pivot point;
wherein the actuator is configured to move the vehicle door between an open position providing access to the interior space and a closed position substantially prohibiting access to the interior space; and
wherein the vehicle door is configured to move along a travel path extending from the open position to the closed position while maintaining an outer surface of the vehicle door substantially parallel to a longitudinal axis of the vehicle.

US Pat. No. 10,564,638

TELEOPERATOR SITUATIONAL AWARENESS

Zoox, Inc., Foster City,...

1. A driverless vehicle comprising:a processor;
a sensor;
a network interface; and
a memory having stored thereon processor-executable instructions that, when executed by the processor, configure the driverless vehicle to:
receive sensor data from at least one of the sensor or the network interface, the sensor data related to operation of the driverless vehicle;
determine a confidence level associated with the operation of the driverless vehicle based at least in part on a portion of the sensor data;
store the confidence level as a stored confidence level and at least the portion of the sensor data as associated sensor data;
transmit, based at least in part on the confidence level, a request for teleoperator assistance via the network interface, the request including the portion of the sensor data and the confidence level;
receive a teleoperations signal via the network interface; and
control the driverless vehicle based at least in part on the teleoperations signal.

US Pat. No. 10,414,336

ACOUSTIC NOTIFICATIONS

Zoox, Inc., Foster City,...

1. A system comprising:an acoustic array disposed on at least one exterior surface of a vehicle, the acoustic array comprising a plurality of speakers configured for multi-channel functionality;
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
determining, based at least in part on a dimension of the vehicle, a pattern for outputting at least a first sound and a second sound via individual speakers of the plurality of speakers;
determining, based at least in part on the pattern, a first signal for emitting the first sound via at least a first speaker of the plurality of speakers;
determining, based at least in part on the pattern, a second signal for emitting the second sound via at least a second speaker of the plurality of speakers, the second speaker being associated with a different channel than the first speaker;
causing, based at least in part on the first signal, the first speaker to emit the first sound; and
causing, based at least in part on the second signal, the second speaker to emit the second sound, wherein the first sound and the second sound, when emitted in the pattern, represent a surface boundary of the vehicle.

US Pat. No. 10,386,836

INTERACTIONS BETWEEN VEHICLE AND TELEOPERATIONS SYSTEM

Zoox, Inc., Foster City,...

1. A method for operating a driverless vehicle comprising a vehicle controller, the method comprising:receiving, at the driverless vehicle, sensor signals comprising sensor data from one or more sensors associated with the driverless vehicle, the sensor data related to operation of the driverless vehicle;
receiving road network data from a road network data store, the road network data being based at least in part on a location of the driverless vehicle;
determining, at the driverless vehicle, a driving corridor within which the driverless vehicle travels according to a trajectory, the driving corridor comprising virtual boundaries and being based at least in part on at least one of the sensor data or the road network data;
causing the driverless vehicle to traverse a road network autonomously according to a path from a first geographic location to a second geographic location different than the first geographic location;
determining that an event associated with the path has occurred;
based at least in part on determining that the event has occurred, sending communication signals from the driverless vehicle to a teleoperations system, the communication signals comprising a request for guidance from the teleoperations system and one or more of a portion of sensor data and the road network data;
receiving, at the driverless vehicle, teleoperations signals from the teleoperations system; and
determining, by the vehicle controller, a revised trajectory based at least in part on the teleoperations signals.

US Pat. No. 10,379,538

TRAJECTORY GENERATION USING MOTION PRIMITIVES

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the one or more processors to:
determine a reference trajectory for an autonomous vehicle to follow in an environment, the reference trajectory based at least in part on a centerline associated with a portion of a road in the environment;
determine one or more segments of the reference trajectory, a segment of the one or more segments having a plurality of points, wherein each point of the plurality of points is associated with a same sign of a curvature value, and wherein the curvature value of the segment varies linearly with respect to an arc length along the reference trajectory;
determine an acceleration value of the autonomous vehicle, wherein the each point of the plurality of points is associated with the acceleration value;
determine a lateral offset of the autonomous vehicle relative to the reference trajectory;
determine, based at least in part on the lateral offset, an upper bound for an arc length associated with the segment;
determine, based at least in part on the lateral offset, a lower bound for the arc length associated with the segment;
define an approximate motion primitive associated with the segment; and
control the autonomous vehicle based, at least in part, on the approximate motion primitive.

US Pat. No. 10,472,004

BI-DIRECTIONAL SPOILER

Zoox, Inc., Foster City,...

1. A vehicle comprising:a body having an outer surface, the outer surface comprising a top surface and an adjoining surface, the top surface and the adjoining surface extending between lateral sides of the vehicle; and
a region of the body configured to alter a drag coefficient of the vehicle, the region comprising:
an indentation in the outer surface proximate a transition from the top surface to the adjoining surface, the indentation comprising:
a first indentation end proximate the transition, the first indentation end extending laterally between opposite indentation sides;
a second indentation end spaced from the first indentation end, the second indentation end extending laterally between the opposite indentation sides; and
an indentation surface extending between the first indentation end and the second indentation end, the indentation surface comprising a first surface extending from the first indentation end at a first angle relative to the outer surface, a second surface extending from the second indentation end at a second angle relative to the outer surface, and a trough disposed between the first surface and the second surface, each of the first surface, the second surface and the trough extending laterally between the opposite indentation sides; and
a foil at least partially disposed in the indentation and extending laterally between the opposite indentation sides, the foil comprising a first foil end, a second foil end spaced from the first foil end, a top foil surface extending between the first foil end and the second foil end and a bottom foil surface spaced from the top foil surface and extending between the first foil end and the second foil end,
wherein the first foil end and the second foil end are displaced between the first indentation end and the second indentation end, and the bottom foil surface is spaced from the indentation surface.

US Pat. No. 10,457,179

IMMERSIVE VEHICLE SEATS

Zoox, Inc., Foster City,...

1. A method of controlling a vehicle seat in a vehicle, the vehicle seat comprising one or more haptic elements configured to provide a haptic output perceptible as a tactile sensation at a surface of the vehicle seat, the method comprising:receiving presence information indicative of an occupant occupying the vehicle seat;
receiving a signal indicative of an input to a user interface, the input comprising a user selection associated with a haptic setting associated with the vehicle seat;
determining an event, the determining the event comprising receiving, via one or more sensors associated with the vehicle, object information indicative of a presence of an object in an environment of the vehicle, and determining an object type associated with the object; and
controlling the one or more haptic elements to produce the haptic output based at least in part on the object type.

US Pat. No. 10,627,818

TEMPORAL PREDICTION MODEL FOR SEMANTIC INTENT UNDERSTANDING

Zoox, Inc., Foster City,...

1. A system associated with predicting an intent of an object in an environment proximate an autonomous vehicle, the system comprising:one or more sensors, wherein the one or more sensors are associated with the autonomous vehicle;
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform actions comprising:
receiving sensor data from the one or more sensors;
detecting, based at least in part on a portion of the sensor data, the object in the environment;
determining, based at least in part on the sensor data, at least one attribute of the object, the at least one attribute comprising a classification distribution of the object indicating individual probabilities that the object is associated with individual classes of a plurality of classes;
generating, based at least in part on the attribute and using a machine learned model, a set of intents;
associating, based at least in part on the attribute, one or more weights with a subset of intents, the subset of intents corresponding to a subset of candidate trajectories;
determining a weight associated with a candidate trajectory of the subset of candidate trajectories meets or exceeds a threshold, the candidate trajectory corresponding to the intent; and
determining a vehicle trajectory along which the autonomous vehicle is to drive based at least in part on the intent of the object.

US Pat. No. 10,606,259

INTERACTIONS BETWEEN VEHICLE AND TELEOPERATIONS SYSTEM

Zoox, Inc., Foster City,...

1. A teleoperations system for assisting with operating a driverless vehicle, the driverless vehicle comprising a vehicle controller and configured to autonomously operate via the vehicle controller along a road network according to a path from a first geographic location to a destination separated from the first geographic location, the teleoperations system comprising:a teleoperations receiver configured to receive communication signals from the driverless vehicle, the communication signals comprising:
at least a portion of sensor data from one or more sensors associated with the driverless vehicle, the at least a portion of sensor data related to operation of the driverless vehicle; and
data indicating occurrence of an event associated with the path, wherein the data indicating occurrence of the event comprises data indicating a confidence level associated with the path is less than a threshold confidence level, wherein the confidence level is determined by the driverless vehicle;
a teleoperations interface configured to facilitate:
reviewing the at least a portion of sensor data and the data indicating occurrence of the event; and
determining a level of guidance to provide the driverless vehicle based at least in part on at least one of the at least a portion of sensor data or the data indicating occurrence of the event; and
a teleoperations transmitter configured to transmit teleoperations signals to the driverless vehicle, the teleoperations signals comprising guidance to alter a virtual boundary of a driving corridor and guidance to operate the driverless vehicle according to the virtual boundary,
wherein the driverless vehicle is configured to maneuver via the vehicle controller to at least one of avoid the event, travel around the event, or pass through the event based at least in part on the teleoperations signals.

US Pat. No. 10,591,910

MACHINE-LEARNING SYSTEMS AND TECHNIQUES TO OPTIMIZE TELEOPERATION AND/OR PLANNER DECISIONS

Zoox, Inc., Foster City,...

1. A method comprising:receiving telemetry data from an autonomous vehicle, the telemetry data comprising sensor data from a sensor on the autonomous vehicle;
receiving policy data;
determining, based at least in part on at least one of the telemetry data or the policy data, an event in a region of an environment through which the autonomous vehicle has traversed;
determining, based at least in part on the event, that a state of operation of the autonomous vehicle is non-normative, the non-normative state indicating an insufficiency of guaranteeing collision-free travel;
determining, based at least in part on the state of operation, a plurality of trajectories;
determining, based at least in part on a trajectory of the plurality of trajectories, updated policy data; and
transmitting the updated policy data to the autonomous vehicle, the updated policy data configured to cause the autonomous vehicle to operate according to one or more policies of the updated policy data.

US Pat. No. 10,315,563

ACOUSTIC NOTIFICATIONS

Zoox, Inc., Foster City,...

1. A system comprising:a sensor;
an acoustic array disposed on an exterior surface of a vehicle, the acoustic array comprising a plurality of speakers configured to output audio;
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving, from the sensor, sensor data indicative of a pedestrian;
determining, based at least in part on the sensor data, a pedestrian position;
determining, based at least in part on the sensor data, a first direction of propagation to emit a first beam of acoustic energy, the first direction of propagation directing the first beam of acoustic energy towards a first position on a first side of the pedestrian position;
causing, at a first time, at least a first portion of the plurality of speakers to emit the first beam of acoustic energy indicative of an alert in the first direction of propagation;
determining, based at least in part on the sensor data, a second direction of propagation to emit a second beam of acoustic energy, the second direction of propagation directing the second beam of acoustic energy towards a second position on a second side of the pedestrian position; and
causing, at a second time, at least a second portion of the plurality of speakers to emit the second beam of acoustic energy indicative of the alert in the second direction of propagation, the first beam of acoustic energy and the second beam of acoustic energy, when perceived by the pedestrian, enabling the pedestrian to localize the vehicle.

US Pat. No. 10,504,282

GENERATING MAPS WITHOUT SHADOWS USING GEOMETRY

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions cause the system to:
obtain a three-dimensional (3D) mesh representing an environment;
receive a plurality of sensor data representing data captured by one or more image capture devices disposed about one or more autonomous vehicles traversing the environment;
receive, from the plurality of sensor data, a plurality of images representing the environment;
map an image of the plurality of images onto the 3D mesh;
determine a first location of a light source illuminating the environment;
project a ray originating from the first location of the light source onto the 3D mesh;
determine, based at least in part on an intersection point between the ray and the 3D mesh and on first temporal information, a candidate shadow region of the 3D mesh with respect to the light source;
determine, based at least in part on a geometry of the 3D mesh, second temporal information associated with a second location of the light source for illuminating the candidate shadow region, wherein the second location is different than the first location;
select an additional image of the plurality of images based at least in part on the second temporal information; and
generate, based at least in part on the candidate shadow region and the additional image, an updated image representing the environment, the updated image corresponding to the image with one or more shadows omitted.

US Pat. No. 10,593,042

PERSPECTIVE CONVERSION FOR MULTI-DIMENSIONAL DATA ANALYSIS

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more non-transitory computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive a three-dimensional dataset captured by at least one LIDAR sensor installed on an autonomous vehicle, the at least one LIDAR sensor associated with a first perspective;
receive segmentation information associated with the three-dimensional dataset, the segmentation information including, for an individual data point of the three-dimensional dataset, a segmentation identifier associated with a particular object;
select a portion of the three-dimensional dataset based at least in part on the segmentation identifier;
extract, as extracted data, the portion of the three-dimensional dataset, wherein the extracted data represents a first width and a first height relative to the first perspective;
perform a principal component analysis on the extracted data to determine at least a first principal component, a second principal component, and a third principal component of the extracted data;
position a rendering plane relative to the extracted data based at least in part on the principal component analysis, wherein an orientation of the rendering plane is associated with a second perspective different than the first perspective, wherein the orientation of the rendering plane is selected such that the extracted data represents a second width and a second height relative to the second perspective, and wherein the second width is greater than the first width;
project, as projected data, the extracted data onto the rendering plane, the projected data representing a two-dimensional representation of the extracted data; and
perform classification on the projected data to determine an object classification associated with the projected data.

US Pat. No. 10,558,224

SHARED VEHICLE OBSTACLE DATA

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
capture first sensor data from one or more sensors, the one or more sensors including one or more LIDAR sensors, one or more cameras, one or more RADAR sensors, one or more SONAR sensors, or one or more GPS sensors, the first sensor data representing a first environment;
receive second sensor data from a remote system, the second sensor data representing a second environment that is separate from the first environment;
determine a validity of the second sensor data based at least in part on a distance between a first location associated with the one or more sensors and a second location associated with the second sensor data;
generate a trajectory for an autonomous vehicle based at least in part on the first sensor data, the second sensor data, and the validity of the second sensor data; and
navigate the autonomous vehicle based at least in part on the trajectory.

US Pat. No. 10,712,750

AUTONOMOUS VEHICLE FLEET SERVICE AND SYSTEM

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
computer-readable memory comprising computer-executable instructions that, when executed by the one or more processors, cause the system to:
receive sensor data from an autonomous vehicle autonomously transiting via an outdoor road network from a first geographic location to a second geographic location via a planned path, the sensor data indicating an event encountered by the autonomous vehicle;
receive a request for guidance from the autonomous vehicle;
generate substantially simultaneously, and based at least in part on the request for guidance, multiple candidate trajectories for the autonomous vehicle to maneuver from a current trajectory, each of the multiple candidate trajectories using an alternate path configured to reduce impact of the event on operation of the autonomous vehicle and return the autonomous vehicle to the planned path;
calculate confidence levels associated with each of the multiple candidate trajectories, the confidence levels being indicative of a respective degree of certainty that a respective candidate trajectory reduces a probability that the event impacts operation of the autonomous vehicle;
select a particular candidate trajectory from among at least a subset of the multiple candidate trajectories to be used as a guided trajectory; and
transmit the guided trajectory from the system to the autonomous vehicle for use in maneuvering the autonomous vehicle from the current trajectory in response to the event.

US Pat. No. 10,486,485

PERCEPTION BASED SUSPENSION CONTROL

Zoox, Inc., Foster City,...

1. A vehicle comprising:a body;
one or more sensors coupled to the body, the one or more sensors comprising a light detection and ranging (LIDAR) sensor;
a plurality of wheels;
a plurality of suspensions, each suspension of the plurality of suspensions being disposed between the body and a respective wheel of the plurality of wheels, each suspension comprising one or more suspension components;
one or more processors; and
one or more non-transitory computer readable media storing computer-executable instructions that, when executed by the one or more processors, cause the vehicle to perform operations comprising:
receiving, from a global map server, at least a portion of a previously computed global height map comprising a representation of a road surface and a global uncertainty associated with a point on the global height map;
determining, based at least in part on data received from the one or more sensors, a location of the vehicle with respect to the global height map;
identifying, based at least in part on the one or more sensors, a local height map comprising a local representation of the road surface and a local uncertainty, the local height map indicative of a deformation in a track of the vehicle and differing, at least in part, from the global height map;
identifying a wheel of the plurality of wheels corresponding to the track;
causing a suspension of the plurality of suspensions coupled to the wheel to adjust one or more suspension components based at least in part on the deformation in the track of the vehicle, the global uncertainty, and the local uncertainty; and
transmitting, to the global map server, the local height map.

US Pat. No. 10,471,953

OCCUPANT AWARE BRAKING SYSTEM

Zoox, Inc., Foster City,...

13. An electronic device comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to:
receive first vehicle data related to a first vehicle, the first vehicle data comprising at least a first maximum acceleration for the first vehicle;
receive second vehicle data related to a second vehicle, the second vehicle data comprising at least a second maximum acceleration for the second vehicle;
determine a first platooning order for the first vehicle and the second vehicle based at least in part on the first maximum acceleration and the second maximum acceleration; and
send a first signal to the second vehicle, the first signal comprising one or more of the first platooning order or the first maximum acceleration and configured to cause the first vehicle and the second vehicle to form a first platoon based at least in part on the first platooning order.

US Pat. No. 10,444,759

VOXEL BASED GROUND PLANE ESTIMATION AND OBJECT SEGMENTATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive a LIDAR dataset representing an environment;
associate the LIDAR dataset with a voxel space, the voxel space including a plurality of voxels;
determine a surface normal vector associated with three-dimensional data associated with an individual voxel of the plurality of voxels;
determine a reference direction, the reference direction corresponding to an orientation of an autonomous vehicle;
determine that the individual voxel is a locally flat voxel based at least in part on the surface normal vector and the reference direction;
determine a ground cluster, the ground cluster representing a drivable surface for the autonomous vehicle, the ground cluster including a plurality of locally flat voxels, the plurality of locally flat voxels including the locally flat voxel;
remove the ground cluster from the voxel space to determine a subset of the voxel space;
cluster adjacent voxels in the subset of the voxel space to determine at least one object;
generate a trajectory for the autonomous vehicle based at least in part on the at least one object; and
navigate the autonomous vehicle based at least in part on the trajectory.

US Pat. No. 10,460,473

CAMERA CALIBRATION SYSTEM

Zoox, Inc., Foster City,...

1. A calibration system comprising:a light source configured to emit a collimated light;
a cradle configured to house a camera;
one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the calibration system to perform operations comprising:
receiving a parameter associated with the camera;
sending a centering signal to adjust the cradle to a centered position;
sending a first calibration signal, wherein the first calibration signal causes the camera to adjust to a first position, the first position based at least in part on the parameter and comprising a first pitch and a first yaw;
sending a first capture signal to the camera to cause the camera to capture a first image of the collimated light at the first position;
sending a second calibration signal, wherein the second calibration signal causes the camera to adjust to a second position, the second position based at least in part on the parameter and comprising a second pitch and a second yaw;
sending a second capture signal to the camera to cause the camera to capture a second image of the collimated light at the second position;
receiving, from the camera, the first image and the second image;
calculating an intrinsic characteristic of a lens of the camera and an error associated with the lens based at least in part on a comparison between a first location of the collimated light in the first image and a first reference location of the collimated light and between a second location of the collimated light in the second image and a second reference location;
determining that the error is below a threshold error; and
storing the intrinsic characteristic of the lens.

US Pat. No. 10,298,910

INFRASTRUCTURE FREE INTRINSIC CALIBRATION

Zoox, Inc., Foster City,...

1. A system comprising:a vehicle configured to traverse through an environment, the vehicle comprising:
a plurality of cameras disposed on a body of the vehicle such that fields of view of at least two of the cameras at least partially overlap;
one or more processors; and
computer-readable media storing instructions executable by the one or more processors, wherein the instructions cause the system to perform operations comprising:
receiving a set of images from the plurality of cameras, the set of images comprising images representing the environment at substantially a first time and captured by the plurality of cameras,
determining, for the set of images, a plurality of point pairs, each of the point pairs comprising a first point in one of the images of the set of images and a second point in another of the images of the set of images, the first point and the second point corresponding to a same image feature;
determining first errors associated with the plurality of point pairs, the first errors comprising, at least in part, a first distance between the first point and an epipolar line corresponding to the second point;
determining, based at least in part on the first errors, a first subset of the plurality of point pairs;
determining, from the first subset of the plurality of point pairs, a first correction function representative of a misalignment of an estimated relative pose of the cameras;
determining second errors associated with the plurality of point pairs, the second errors comprising, at least in part, a second distance between the first point and a re-projection of the second point estimating a point depth;
determining, based at least in part on the second errors, a second subset of the plurality of point pairs;
determining, from the second subset of the plurality of point pairs, a second correction function representative of a misalignment of an estimated relative pose of the cameras;
calibrating the plurality of cameras, based at least in part on the first correction function and the second correction function.

US Pat. No. 10,268,191

PREDICTIVE TELEOPERATOR SITUATIONAL AWARENESS

Zoox, Inc., Foster City,...

1. A teleoperation system comprising:at least one processor; and
at least one memory having stored thereon processor-executable instructions that, when executed by the system, configure the teleoperation system to:
obtain via a network interface a first request for teleoperator assistance from a first driverless vehicle and at least one of first operation state data or first sensor data from the first driverless vehicle;
obtain teleoperator input via a user interface;
transmit a first guidance signal to the first driverless vehicle via the network interface based at least in part on the teleoperator input;
store as historical teleoperation data at least part of the teleoperator input and a first descriptor, the first descriptor determined based at least part on at least one of the first request, the first operation state data, or the first sensor data;
obtain via the network interface a second request for teleoperator assistance from a second driverless vehicle and at least one of second operation state data or second sensor data;
determine a second descriptor based at least in part on the second request, the second operation state data, or the second sensor data;
determine a distance between the first descriptor and the second descriptor in a descriptor space; and
perform an action based at least in part on the distance.

US Pat. No. 10,759,416

INDEPENDENT CONTROL OF VEHICLE WHEELS

Zoox, Inc., Foster City,...

1. A wheel control system configured to facilitate control of a vehicle having a longitudinal axis extending along a line between a first end of the vehicle and a second end of the vehicle opposite the first end of the vehicle, the wheel control system comprising:a first steering assembly configured to be coupled proximate to the first end of the vehicle and control one or more parameters of first wheels coupled proximate to the first end of the vehicle;
a second steering assembly configured to be coupled proximate to the second end of the vehicle and control one or more parameters of second wheels coupled proximate to the second end of the vehicle; and
a wheel controller configured to:
receive a signal indicative of a vehicle maneuver, the signal indicative of a vehicle maneuver comprising a signal indicative of a speed of the vehicle; and
control operation of the first steering assembly to the one or more parameters of the first wheels to pivot the first wheels at first steering angles in a first direction relative to the longitudinal axis of the vehicle and the second steering assembly to control the one or more parameters of the second wheels to pivot the second wheels at second steering angles in a second direction relative to the longitudinal axis of the vehicle according to a first mode based at least in part on the signal indicative of the speed being less than a threshold speed and according to a second mode when the signal indicative of the speed being greater than the threshold speed, the first and second steering angles having opposite signs,
wherein the one or more parameters of the first and second steering assemblies comprise a steering angle and one or more of a caster angle, a camber angle, a toe angle, or a wheel speed.

US Pat. No. 10,468,062

DETECTING ERRORS IN SENSOR DATA

Zoox, Inc., Foster City,...

4. A method, comprising:receiving, with one or more processors, a first signal from a first sensor, the first signal including first sensor data representing an environment;
determining, with the one or more processors and based at least in part on the first sensor data, a first parameter associated with a first group of objects;
receiving, with the one or more processors, a second signal from a second sensor, the second signal including second sensor data representing the environment;
determining, with the one or more processors and based at least in part on the second sensor data, a second parameter associated with a second group of objects;
determining, with the one or more processors and based at least in part on the first sensor data and the second sensor data, a third parameter associated with a third group of objects;
comparing, with the one or more processors, the first parameter or the second parameter to the third parameter to identify a difference between the third parameter and the first parameter or the second parameter; and
at least one of:
initiating, with the one or more processors, a response based at least in part on identifying the difference, or
training, based at least in part on the first signal or the second signal, a machine learning system in communication with the one or more processors.

US Pat. No. 10,414,395

FEATURE-BASED PREDICTION

Zoox, Inc., Foster City,...

1. A system comprising:a processor; and
non-transitory computer-readable media storing instructions executable by the processor, wherein the instructions, when executed by the processor, cause the processor to perform actions comprising:
receiving sensor data from a sensor on an autonomous vehicle;
detecting a vehicle in the sensor data;
associating the vehicle with a lane identification;
determining, based at least in part on a change of the lane identification of the vehicle, the occurrence of a cut-in event, the cut-in event having an associated time;
determining sets of features associated with the cut-in event, the sets of features being associated with a period of time that begins before the event and ends after the event;
aggregating the sets of features with additional sets of features associated with additional cut-in events to generate training data;
training, based at least in part on the training data, a machine learned model for predicting cut-in events; and
sending the machine learned model to the autonomous vehicle for predicting the cut-in events by the autonomous vehicle.

US Pat. No. 10,642,275

OCCULSION AWARE PLANNING AND CONTROL

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors that cause the system to perform operations comprising:
controlling an autonomous vehicle to traverse to a first location in an environment;
capturing, as sensor data, at least LIDAR data or image data of the environment using a sensor of the autonomous vehicle;
determining, based at least in part on accessing map data or on identifying an obstacle in the environment, an occlusion grid comprising a plurality of occlusion fields, an occlusion field of the occlusion grid indicating an occlusion state and an occupancy state of a corresponding region of the environment;
determining, as an occluded region and based at least in part on the sensor data, that at least a first portion of the occlusion grid is occluded at a first time;
determining, as an un-occluded region and based at least in part on the sensor data, that at least a second portion of the occlusion grid is un-occluded at the first time;
determining, based at least in part on an extent and occupancy of the second portion, a confidence level associated with safely traversing a trajectory through the environment;
controlling the autonomous vehicle to stay at the first location or to traverse to a second location based at least in part on the confidence level being below a threshold level;
determining, based at least in part on the sensor data, that a third portion of the occlusion grid is visible and unoccupied at a second time; and
controlling the autonomous vehicle based at least in part on the third portion of the occlusion grid being visible and unoccupied at the second time.

US Pat. No. 10,459,444

AUTONOMOUS VEHICLE FLEET MODEL TRAINING AND TESTING

Zoox, Inc., Foster City,...

1. A vehicle comprising:a vehicle controller operably connected to at least one sensor, a network interface, and a drive system to control operation of the vehicle, the vehicle controller being configured to:
receive an indication that the vehicle controller has unallocated computational resources;
receive a command to utilize the unallocated computational resources, the command causing the vehicle controller to be further configured to conduct one or more of:
testing an experimental machine learning (ML) model based at least in part on causing the drive system to control operation of the vehicle using the experimental ML model, the testing yielding a test result, or
training a target ML model to create a trained ML model, based at least in part on causing the drive system to control operation of the vehicle using the target ML model.

US Pat. No. 10,691,127

TRAJECTORY GENERATION USING TEMPORAL LOGIC AND TREE SEARCH

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
a computer-readable medium storing computer-executable instructions that, when executed, cause the system to perform operations comprising:
generating a plurality of candidate trajectories for an autonomous vehicle;
selecting, based at least in part on an environment proximate to the autonomous vehicle, a temporal logic formula;
evaluating a candidate trajectory of the plurality of candidate trajectories based at least in part on an automaton determined by the temporal logic formula;
selecting, as a selected trajectory, the candidate trajectory based at least part on a result of the evaluating and on a cost associated with the candidate trajectory; and
controlling the autonomous vehicle based at least in part on the selected trajectory.

US Pat. No. 10,489,529

SCENARIO DESCRIPTION LANGUAGE

Zoox, Inc., Foster City,...

1. A computer-implemented method for generating a scenario in a simulation environment, the computer-implemented method comprising:receiving first instructions defining at least a first primitive that is to be instantiated in the scenario, the first primitive corresponding to an autonomous vehicle;
receiving second instructions associated with one or more values to be assigned to the first primitive;
creating, based at least in part on the one or more values, a set of permutations of the first primitive, each permutation of the set of permutations associated with a different value of the one or more values;
instantiating, based at least in part on semantic coordinates indicative of a map topology, each of the permutations within the map topology to generate a set of scenarios, each scenario of the set of scenarios corresponding to a permutation of the set of permutations and the map topology comprising an intersection;
receiving sequence instructions defining at least one of a first condition or a first behavior associated with the first primitive at a first time and at least one of a second condition or a second behavior associated with the first primitive at a second time; and
determining, based at least in part on executing at least a portion of the sequence instructions, information associated with an individual scenario of the set of scenarios, the information describing how an autonomous controller associated with the autonomous vehicle responds given the individual scenario generated in the simulation environment.

US Pat. No. 10,317,897

WEARABLE FOR AUTONOMOUS VEHICLE INTERACTION

Zoox, Inc., Foster City,...

1. A wearable device for enabling passenger interaction with an autonomous vehicle comprising:a transceiver;
a sensor, an output of which comprises a passenger input signal;
a notification element, the notification element comprising at least one of a vibration element;
one or more processors in communication with the transceiver, the sensor, and the notification element; and
memory storing computer-executable instructions that, when executed by the one or more processors, cause the wearable device to:
receive a passenger communication signal from the autonomous vehicle;
activate the notification element based at least in part on the passenger communication signal; and
send the passenger input signal to the autonomous vehicle,
wherein the passenger input signal comprises passenger preference information, the passenger preference information comprising at least one of specified vehicle lighting levels, specified vehicle audio levels, specified music to be played in the autonomous vehicle, specified routes for the autonomous vehicle to follow, or specified ride profiles,
wherein sending the passenger input signal to the autonomous vehicle causes the autonomous vehicle to operate according to the passenger preference information,
wherein the instructions, when executed, further cause the one or more processors to determine a hand gesture based at least in part on the passenger input signal and, based at least in part on the hand gesture, at least one of summon a ride from the autonomous vehicle or cancel a ride from the autonomous vehicle; and
cause the notification element to vibrate a first number of times when the autonomous vehicle is a first distance from the wearable device and vibrate a second number of times when the autonomous vehicle is a second distance from the wearable device, the first number of times and the second number of times differing from one another, and the first distance and the second distance differing from one another.

US Pat. No. 10,733,441

THREE DIMENSIONAL BOUNDING BOX ESTIMATION FROM TWO DIMENSIONAL IMAGES

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
a non-transitory computer readable medium comprising instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving sensor data;
determining an object in an environment represented in the sensor data;
inputting at least a portion of the sensor data into a machine learning algorithm;
receiving, based at least in part on the portion of the sensor data and from the machine learning algorithm, output associated with a physical parameter of the object,
wherein the machine learning algorithm comprises:
a coarse output branch configured to output a coarse output; and
a fine offset branch configured to output an offset with respect to the coarse output by the coarse output branch; and
wherein the output comprises a sum of the offset and a highest confidence value of a set of confidence values associated with the coarse output.

US Pat. No. 10,438,371

THREE-DIMENSIONAL BOUNDING BOX FROM TWO-DIMENSIONAL IMAGE AND POINT CLOUD DATA

Zoox, Inc., Foster City,...

1. A system for estimating a three-dimensional bounding box, the system including a non-transitory computer readable medium containing instructions that, when executed by one or more processors, cause the system to:receive an image captured from an image capture device;
detect an object in the image;
crop the image to form a cropped image including the object;
receive point cloud data associated with the object;
determine, using a first processing algorithm, a first feature vector associated with the point cloud data, the first feature vector comprising a geometric feature vector associated with one or more locations of one or more points in the point cloud data;
determine, using a second processing algorithm, a second feature vector associated with the cropped image, the second feature vector comprising an appearance feature vector;
pass the first feature vector and the second feature vector into a neural network; and
receive, from the neural network, coordinates descriptive of a three-dimensional bounding box associated with the object.

US Pat. No. 10,353,390

TRAJECTORY GENERATION AND EXECUTION ARCHITECTURE

Zoox, Inc., Foster City,...

1. A system embodied in an autonomous vehicle, the system comprising:a first computer system comprising:
one or more first processors; and
one or more first computer readable storage media communicatively coupled to the one or more first processors and storing one or more first modules including instructions that are executable by the one or more first processors, the one or more first modules comprising:
a route planning module configured to generate a route between a current location of the autonomous vehicle and a target location of the autonomous vehicle, the route planning module operating at a first operational frequency;
a decision module configured to generate an instruction based on the route, the instruction including a sequence of actions to guide the autonomous vehicle along at least a portion of the route, the decision module operating at a second operational frequency; and
a trajectory module configured to:
access real-time processed sensor data; and
generate an output trajectory based at least in part on the instruction and the real-time processed sensor data, the trajectory module operating at a third operational frequency; and
a second computer system comprising:
one or more second processors; and
one or more second computer readable storage media communicatively coupled to the one or more second processors and storing one or more second modules including instructions that are executable by the one or more second processors, the one or more second modules comprising an execution module configured to generate one or more signals for causing the autonomous vehicle to drive along the output trajectory, the execution module operating at a fourth operational frequency.

US Pat. No. 10,733,482

OBJECT HEIGHT ESTIMATION FROM MONOCULAR IMAGES

Zoox, Inc., Foster City,...

1. A system comprising: one or more processors; and one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to: receive an image from an image capture device; detect an object in the image; output a region of interest associated with the object detected in the image; crop the image at the region of interest to create a cropped image; resize and scale the cropped image to a predefined size to create a resized and scaled image; transmit the resized and scaled image into a convolutional neural network; and determine, based at least upon an output of the convolutional neural network, an estimated object size, wherein the convolutional neural network outputs a probability distribution of sizes, the probability distribution represented by more than one bin, each bin representing a range of sizes, and further wherein the estimated object size is based, at least in part, on the probability distribution.

US Pat. No. 10,688,841

EXPANDING SENSOR DOMAIN COVERAGE USING DIFFERENTIAL ACTIVE SUSPENSION

Zoox, Inc., Foster City,...

1. A method comprising:receiving, at a controller, sensor data from one or more sensors on a vehicle, the vehicle having a first vehicle pose;
receiving, at the controller, perception data indicating that the sensor data contains a first portion of an object of interest, the object of interest being at least partially outside a field of view of a sensor of the one or more sensors on the vehicle, the sensor having a fixed field of view relative to the vehicle;
determining, with the controller, a second vehicle pose, such that in the second vehicle pose a second portion of the object of interest is within the field of view of the sensor, the second portion being different than the first portion;
receiving additional perception data comprising an object classification of an interior object;
determining, based at least in part on the object classification, one or more of a maximum tilt angle or maximum tilt rate of the vehicle; and
sending, from the controller, a signal to a suspension controller to cause the vehicle to assume the second vehicle pose, wherein the signal comprises one or more instructions to constrain the second vehicle pose based at least in part on the maximum tilt angle or instructions to constrain motion of a suspension based at least in part on a maximum tilt rate.

US Pat. No. 10,682,921

UNDERBODY CHARGING OF VEHICLE BATTERIES

Zoox, Inc., Foster City,...

1. A charge coupler comprising:an electrical connector configured to be contacted and coupled to an electrical power source;
one or more electrical contacts configured to be electrically coupled to one or more electrical contacts of a vehicle from under the vehicle;
a receiver configured to receive a signal from a transmitter associated with the vehicle for activating the charge coupler and electrically connecting the electrical power source to the one or more electrical contacts of the vehicle based at least in part on a distance between the receiver and the transmitter, the receiver comprising an inductive coupling configured to receive, wirelessly, an amount of electrical power and the signal comprising a power transmission; and
wherein, based at least in part on the signal, the electrical power source is electrically coupled to the one or more electrical contacts of the charge coupler, such that the electrical power source and the one or more electrical contacts of the vehicle are electrically coupled to one another upon contact between the one or more electrical contacts of the charge coupler and the one or more electrical contacts of the vehicle.

US Pat. No. 10,678,740

COORDINATED COMPONENT INTERFACE CONTROL FRAMEWORK

Zoox, Inc., Foster City,...

1. A method comprising:receiving sensor data from a sensor on an autonomous vehicle;
processing the sensor data by a first system, wherein the first system comprises a first component and one or more other components and wherein the processing is based at least in part on controlling operation of the first component and the one or more other components;
recording a sequence of events based at least in part on receiving one or more messages from a first controller associated with the first component during the processing the sensor data by the first system;
receiving, based at least in part on processing the sensor data by the first system, a first output;
replacing the first component with a second component to obtain a second system comprising the second component and the one or more other components;
processing the sensor data by the second system based at least in part on causing the second component to operate according to the sequence;
receiving, based at least in part on processing the sensor data by the second system, a second output; and
transmitting, to the autonomous vehicle, instructions to replace the first component with the second component based at least in part on determining a difference between the first output and the second output.

US Pat. No. 10,671,075

TRAJECTORY GENERATION USING CURVATURE SEGMENTS

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the one or more processors to:
determine a reference trajectory for an autonomous vehicle to traverse an environment;
determine a first frame associated with the reference trajectory;
determine a second frame associated with the reference trajectory, the second frame associated with a tangent vector of a second point on the reference trajectory;
determine a state of the autonomous vehicle including at least a position of the autonomous vehicle relative to the reference trajectory;
determine a candidate trajectory of the autonomous vehicle, the candidate trajectory including a segment having an origin and a curvature value associated with the position of the autonomous vehicle and intersecting at an intersection point with a reference line that is substantially perpendicular to the tangent vector associated with the second point, the intersection point based at least in part on the curvature value; and
output an output trajectory to a vehicle controller to control the autonomous vehicle, the output trajectory based at least in part on the candidate trajectory.

US Pat. No. 10,671,076

TRAJECTORY PREDICTION OF THIRD-PARTY OBJECTS USING TEMPORAL LOGIC AND TREE SEARCH

Zoox, Inc., Foster City,...

1. A system of an autonomous vehicle, the system comprising:one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive sensor data from one or more of a LIDAR sensor, a RADAR sensor, or a camera sensor;
determine, based at least in part on the sensor data, one or more dynamic symbols, a dynamic symbol of the one or more dynamic symbols representing a third-party object that is not under control of the autonomous vehicle;
determine, based at least in part on the sensor data, motion data associated with the one or more dynamic symbols, the motion data including at least a position, an orientation, and a velocity associated with a dynamic symbol of the one or more dynamic symbols;
determine, based at least in part on the one or more dynamic symbols and one or more static symbols, one or more predictive trajectories navigable by the third-party object, wherein the one or more dynamic symbols and the one or more static symbols represent temporal logic features;
determine, based at least in part on the motion data, an outcome probability associated with a predictive trajectory of the one or more predictive trajectories;
select, as a selected predictive trajectory, the predictive trajectory based at least in part on the outcome probability; and
provide the selected predictive trajectory to a decision planner of the autonomous vehicle.

US Pat. No. 10,649,459

DATA SEGMENTATION USING MASKS

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
capturing sensor data using a light detection and ranging (LIDAR) sensor on a vehicle;
associating the sensor data with a voxel space, the voxel space comprising at least three dimensions;
generating a two-dimensional representation of the voxel space, the two-dimensional representation comprising a number of image channels;
inputting the two-dimensional representation into a machine learning algorithm;
receiving, from the machine learning algorithm, a first mask representing an object in the voxel space in two dimensions;
generating, based at least in part on an expansion of the first mask, a second mask, the expansion based at least in part on a region growing algorithm, a size of the first mask, or an intersection with a third mask associated with another object; and
segmenting, based at least in part on the second mask, the sensor data.

US Pat. No. 10,613,548

ARCHITECTURE FOR SECURE VEHICLE CONTROL

Zoox, Inc., Foster City,...

1. An autonomous vehicle system comprising:an autonomous vehicle;
an interface configured to receive a request associated with the autonomous vehicle system;
a transmitter on the autonomous vehicle;
a receiver on the autonomous vehicle;
a first computer system on the autonomous vehicle, the first computer system comprising:
one or more first processors; and
one or more first non-transitory computer readable storage media communicatively coupled to the one or more first processors and storing one or more first instructions that are executable by the one or more first processors, execution of the one or more first instructions causing the one or more first processors to perform acts comprising:
determining first information based at least in part on the request; and
transmitting, via the transmitter, the first information to a remote computing system, the remote computing system being configured to store the first information in a first remote database, to determine an action responsive to the request, and to cause the action to be stored in a second remote database; and
a second computer system on the autonomous vehicle, the second computer system comprising:
one or more second processors; and
one or more second non-transitory computer readable storage media communicatively coupled to the one or more second processors and storing one or more second instructions that are executable by the one or more second processors, execution of the one or more second instructions causing the one or more second processors to perform acts comprising:
querying the second remote database to generate a query response;
receiving via the receiver, based at least in part on the query response, second information indicative of the action responsive to the request; and
controlling an aspect of the autonomous vehicle to implement the action based at least in part on the second information.

US Pat. No. 10,409,284

SYSTEM OF CONFIGURING ACTIVE LIGHTING TO INDICATE DIRECTIONALITY OF AN AUTONOMOUS VEHICLE

Zoox, Inc., Foster City,...

1. A method comprising:causing an autonomous vehicle to navigate in a first direction of travel within an environment, the autonomous vehicle configured to navigate in the first direction of travel in which a first end of the autonomous vehicle is leading and a second end of the autonomous vehicle is trailing and a second direction of travel in which the second end is leading and the first end is trailing;
identifying a location of an object within the environment;
determining an orientation of the autonomous vehicle relative to the location of the object;
selecting, based at least in part on the orientation of the autonomous vehicle, a light emitter of the autonomous vehicle to emit a light pattern to visually convey the first direction of travel; and
causing the light emitter to emit the light pattern to visually convey the first direction of travel.

US Pat. No. 10,964,349

DETECTING ERRORS IN SENSOR DATA

Zoox, Inc., Foster City,...

4. A method comprising:receiving first sensor data;
determining, based at least in part on the first sensor data, a first indication of a first object in the environment;
receiving second sensor data;
determining, based at least in part on the first sensor data and the second sensor data, a second indication of a second object in the environment;
determining a difference between the first indication and the second indication; and
at least one of:
initiating a response based at least in part on the difference, or
outputting, based at least in part on the first sensor data and the second sensor data, an indication to train a machine learning system.

US Pat. No. 10,745,003

RESILIENT SAFETY SYSTEM FOR A ROBOTIC VEHICLE

Zoox, Inc., Foster City,...

1. A method for controlling an autonomous vehicle configured to drive autonomously on a roadway in an environment, the autonomous vehicle having a plurality of sensors, the method comprising:calculating, based at least in part on a location of the autonomous vehicle in the environment, a current trajectory of the autonomous vehicle;
receiving first sensor data from one or more of the plurality of sensors;
detecting, based at least in part on a portion of the first sensor data, an object in the environment;
generating, based at least in part on the first sensor data and the location of the object, a predicted object trajectory of the object in the environment;
determining, based at least in part on the current trajectory and the predicted object trajectory, a first time to impact of the object with the autonomous vehicle at an intersection of the current trajectory and the predicted object trajectory;
determining, based at least in part on the first time to impact being less than or equal to a first threshold time, a safety output comprising an alert configured to be perceptible by the object;
outputting, using a safety system, the safety output;
receiving second sensor data from the one or more of the plurality of sensors;
generating, based at least in part on the second sensor data, an updated predicted object trajectory;
determining, based at least in part on the current trajectory and the updated predicted object trajectory, an updated time to impact of the object with the autonomous vehicle;
calculating, based at least in part on the updated time to impact being equal to or below a second threshold time lower than the first threshold time, a maneuver for the autonomous vehicle to perform; and
causing the autonomous vehicle to perform the maneuver.

US Pat. No. 10,737,652

CRASH SENSING USING CAVITY PRESSURE

Zoox, Inc., Foster City,...

15. A vehicle comprising:a body having a leading surface and a trailing surface, the leading surface and the trailing surface being separated by a first distance in a longitudinal direction;
a leading tire, at least a portion of the leading tire disposed, in the longitudinal direction, a second distance from the trailing surface, the second distance being greater than the first distance of the leading surface from the trailing surface;
a pressure sensor disposed to sense a pressure associated with the leading tire;
a safety system; and
a safety system controller communicatively coupled to the pressure sensor and the safety system, the safety system controller being configured to:
receive information from the pressure sensor indicative of an increase in pressure resulting from contact with an object;
determine, from perception information, one or more attributes of the object;
determine, based at least in part on the one or more attributes and the perception information, whether to activate the safety system.

US Pat. No. 10,733,761

SENSOR CALIBRATION

Zoox, Inc., Foster City,...

1. A system comprising:a vehicle configured to traverse through an environment, the vehicle comprising:
a plurality of cameras disposed on a body of the vehicle such that fields of view of at least two of the cameras at least partially overlap;
one or more processors; and
computer-readable media storing instructions executable by the one or more processors, wherein the instructions cause the system to perform operations comprising:
receiving a set of images from the plurality of cameras, the set of images comprising images representing the environment captured by the plurality of cameras,
determining, for the set of images, a plurality of point pairs, each of the point pairs comprising a first point in one of the images of the set of images and a second point in another of the images of the set of images, the first point and the second point corresponding to a same image feature;
determining errors associated with the plurality of point pairs, the errors comprising, at least in part, a distance between the first point and an epipolar line corresponding to the second point;
determining, based at least in part on the errors, a subset of the plurality of point pairs;
determining, from the subset of the plurality of point pairs, a correction function representative of a misalignment of an estimated relative pose of the cameras; and
calibrating the plurality of cameras, based at least in part on the correction function.

US Pat. No. 10,703,323

OCCUPANT PROTECTION SYSTEM INCLUDING EXPANDABLE CURTAIN AND/OR EXPANDABLE BLADDER

Zoox, Inc., Foster City,...

1. An occupant protection system for a vehicle, the occupant protection system comprising:an expandable curtain configured to expand from a stowed state to a deployed state having a length configured to extend at least a portion of a distance between a vehicle roof and a vehicle floor, wherein the expandable curtain in the deployed state comprises:
a first side configured to extend in a first direction along a portion of a first interior side of the vehicle;
a second side spaced from the first side and configured to extend in the first direction along a portion of a second interior side of the vehicle, the second interior side being spaced and opposite from the first interior side; and
a transverse portion extending in a second direction substantially perpendicular to the first direction and between the first side and the second side of the expandable curtain, the transverse portion comprising an expandable support chamber substantially centered in the transverse portion, the first side of the expandable curtain, the transverse portion, and the second side of the expandable curtain being contiguous and extending from the first side of the expandable curtain to the second side of the expandable curtain; and
a first expandable bladder configured to expand from a stowed state to a deployed state, such that the expandable bladder in the deployed state nests between the first side and the expandable support chamber of the expandable curtain.

US Pat. No. 10,699,477

GENERATING MAPS WITHOUT SHADOWS

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein executing the instructions causes the system to:
receive a plurality of sensor data captured by one or more sensors disposed about autonomous vehicles traversing an environment;
obtain, from the sensor data, a plurality of images representing the environment;
determine, based at least in part on a machine learning algorithm trained to identify one or more shadow regions in image data, a candidate shadow region in an image of the plurality of images;
determine a corresponding region in one or more additional images of the plurality of images;
provide the candidate shadow region and the corresponding region to a filtering algorithm;
generate, using the filtering algorithm and based at least in part on the candidate shadow region and the corresponding region, an updated region representing the candidate shadow region, wherein generating the updated region comprises generating updated region image data having a contribution of one or more shadows represented in the candidate shadow region minimized or omitted;
generate, based at least in part on the updated region, a composite image including at least a portion of the image and the updated region; and
map the image or the composite image onto a three-dimensional mesh associated with the environment.

US Pat. No. 10,427,631

VEHICULAR ADAPTATION FOR IMPACT MITIGATION

Zoox, Inc., Foster City,...

1. A vehicle comprising:a body;
a sensor coupled to the body to sense a condition;
an adaptive structure coupled to the body to adjust a rigidity of the body;
one or more processors; and
one or more non-transitory computer readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving input from the sensor of the vehicle;
determining that a value of the input exceeds a threshold value associated with a parameter; and
modifying the rigidity of the body by actuating the adaptive structure of the vehicle, based at least in part on the input, wherein the modifying the rigidity of the vehicle is based at least in part on the determining that the value exceeds the threshold value.

US Pat. No. 11,110,918

DYNAMIC COLLISION CHECKING

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:receiving a vehicle trajectory associated with an autonomous vehicle traversing an environment;
receiving sensor data from a sensor associated with the autonomous vehicle;
determining an object represented in the sensor data;
determining an object type associated with the object;
determining a drivable area on which the autonomous vehicle is to navigate;
determining, based at least in part on the object type, a virtual boundary of the drivable area;
determining a point on the vehicle trajectory associated with a portion of the vehicle trajectory;
determining, based at least in part on the point on the vehicle trajectory, the virtual boundary of the drivable area, and a width associated with the autonomous vehicle, a cost;
determining, based at least in part on the cost, an action of the autonomous vehicle associated with the point on the vehicle trajectory; and
controlling the autonomous vehicle based at least in part on the action.


US Pat. No. 10,884,614

ACTUATION INTERFACE

Zoox, Inc., Foster City,...

1. A vehicle comprising:a cover movably coupled to a body of the vehicle configured to move between a closed position in which the cover covers an opening in a compartment of the vehicle and an open position in which the cover partially covers the opening;
one or more processors and memory, the memory storing instructions that, when executed by the one or more processors, configure the vehicle to perform operations comprising:
displaying, at a touch interface, two or more selectable controls associated with actuating the opening, wherein a selectable control of the two or more selectable controls comprises at least a first indicator displayed within the selectable control and a second indicator displayed within the selectable control;
receiving, at the touch interface, touch input from a user;
detecting, via at least the first indicator of the selectable control displayed on the touch interface, a number of concurrent touch inputs that are received at the touch interface;
determining whether to actuate the cover to move based at least in part on the number of the concurrent touch inputs that are received via the touch interface, wherein:
responsive to detecting that the number of concurrent touch inputs received via the selectable control is one, causing;
feedback to be displayed on the touch interface indicating that the touch input requires an additional touch input to select the selectable control, the feedback comprising highlighting at least the second indicator; and
the cover to remain at a current position; and
responsive to detecting that the number of concurrent touch inputs received via the selectable control is more than one;
causing the selectable control to be selected; and
actuating the cover to move based at least in part on a duration or a distance associated with the touch input.

US Pat. No. 10,823,353

MULTIFUNCTION LIGHTING UNIT

Zoox, Inc., Foster City,...

1. A light unit configured to operate as a headlight or a tail light for a vehicle, the light unit comprising:a headlight light source;
a first light source corresponding to a backup light;
a second light source corresponding to a brake light;
first optics positioned to be illuminated by the first light source and second optics positioned to be illuminated by the second light source; and
a lens array positioned to receive light from the first light source and the second light source, the lens array including:
a first portion having a first set of lenses having first characteristics to operate as the first optics; and
a second portion having a second set of lenses having second characteristics to operate as the second optics.

US Pat. No. 10,809,081

USER INTERFACE AND AUGMENTED REALITY FOR IDENTIFYING VEHICLES AND PERSONS

Zoox, Inc., Foster City,...

16. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising:sending, by a user device, a request for data associated with a vehicle;
receiving, at the user device, mission data comprising an indication of a pickup location, wherein the pickup location is determined based in part on a perception system associated with the vehicle and one or more of sensor data or route data associated with the vehicle;
capturing, by the user device, first image data representing an environment;
generating, based at least in part on the first image data and the indication of the pickup location, second image data comprising at least a portion of the first image data and a computer-generated element; and
displaying, by the user device, the second image data.

US Pat. No. 10,737,737

VEHICLE WITH INTERCHANGEABLE DRIVE MODULES

Zoox, Inc., Foster City,...

1. A vehicle comprising:a body module having a first end and a second end, the body module comprising:
a passenger compartment to house one or more passengers; and
a vehicle computing device to control operation of the vehicle;
a drive module removably coupled to the body module at the first end of the body module,
wherein the drive module comprises:
a drive module frame;
first and second wheels;
a propulsion system coupled to the drive module frame and to the first and second wheels to drive at least one of the first and second wheels; and
a crash structure coupled to the drive module frame and comprising one or more brackets to couple the crash structure to the body module of the vehicle, the crash structure configured to absorb an impact force imparted to the first end of the vehicle.

US Pat. No. 10,580,158

DENSE DEPTH ESTIMATION OF IMAGE DATA

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the one or more processors to:
receive image data captured by at least one image sensor of an autonomous vehicle, the image data including a plurality of pixels;
determine a depth estimate associated with a pixel of the plurality of pixels using a machine learned algorithm, the depth estimate providing an indication of a distance between the at least one image sensor and an object represented by the pixel;
determine a confidence value associated with the depth estimate using the machine learned algorithm; and
generate a trajectory for the autonomous vehicle based at least in part on the image data, the depth estimate, and the confidence value,
wherein the machine learned algorithm is trained based at least in part by:
receiving a first plurality of data points;
receiving ground truth data captured by at least one depth sensor of the autonomous vehicle, the ground truth data including a second plurality of data points, wherein the second plurality of data points comprises fewer data points than the first plurality of data points;
clustering the ground truth data into a plurality of clusters; and
determining a first portion of clusters as input depth data for training, wherein the first portion of clusters represents a subset of the ground truth data.

US Pat. No. 11,062,461

POSE DETERMINATION FROM CONTACT POINTS

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the one or more processors to:receive image data captured by at least one image sensor, the image data including a representation of an object;
determine a plurality of object contact points in the image data associated with the object, the plurality of object contact points representing respective points of contact between respective portions of the object and a ground surface;
determine an intersection of a ray associated with an object contact point of the plurality of object contact points with a three-dimensional surface mesh to determine an estimated location of the object contact point with respect to the three-dimensional surface mesh;
determine a three-dimensional bounding box associated with the object based at least in part on the intersection; and
generate a trajectory for a vehicle to follow based at least in part on the three-dimensional bounding box associated with the object.


US Pat. No. 11,042,153

ADAPTIVE MULTI-NETWORK VEHICLE ARCHITECTURE

Zoox, Inc., Foster City,...


1. A system of an autonomous vehicle, the system comprising:a first modem;
a second modem;
a sensor;
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising:determining first network parameters associated with a first wireless network;
determining second network parameters associated with a second wireless network;
capturing, using the sensor, sensor data of an environment, at least a first portion of the sensor data associated with a first priority, wherein the first portion of the sensor data comprises image data;
determining, based at least in part on the sensor data, vehicle data, at least a second portion of the vehicle data associated with a second priority, wherein the second portion of the vehicle data comprises an object represented in the image data;
determining, based at least in part on the first network parameters, the second network parameters, and the first priority, the first portion of the sensor data to transmit via the first modem associated with a first transmission queue, wherein the first transmission queue is a higher priority queue;
determining, based at least in part on the first network parameters, the second network parameters, and the second priority, the second portion of the vehicle data to transmit via the second modem associated with a second transmission queue, wherein the second transmission queue is a lower priority queue;
transmitting, via the first modem and the first transmission queue and to a remote computing device, the first portion of the sensor data;
transmitting, via the second modem and the second transmission queue and to the remote computing device, the second portion of the vehicle data;
receiving, via at least one of the first modem or the second modem, a command from the remote computing device; and
controlling the autonomous vehicle based at least in part on the command.


US Pat. No. 11,023,749

PREDICTION ON TOP-DOWN SCENES BASED ON ACTION DATA

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving sensor data of an environment captured by a sensor of an autonomous vehicle;
receiving action data associated with a candidate action for the autonomous vehicle to perform in the environment, wherein the candidate action comprises one or more of a stay-in-lane action, a turn action, or a lane change action, and wherein the action data is indicative of the candidate action;
generating, based at least in part on the sensor data and the action data, a multi-channel image representing a top-down view of the environment, the multi-channel image representing a bounding box associated with a vehicle separate from the autonomous vehicle in the environment proximate the autonomous vehicle, one or more of kinematic information or semantic information associated with the vehicle, and the candidate action associated with the autonomous vehicle;
inputting the multi-channel image into a machine learned model trained to generate a heat map comprising a prediction probability of a possible location associated with the vehicle;
determining, based at least in part on the heat map, a cost associated with the candidate action; and
determining, based at least in part on the cost associated with the candidate action, a trajectory for the autonomous vehicle to travel in the environment.

US Pat. No. 10,984,290

MULTI-TASK LEARNING FOR REAL-TIME SEMANTIC AND/OR DEPTH AWARE INSTANCE SEGMENTATION AND/OR THREE-DIMENSIONAL OBJECT BOUNDING

Zoox, Inc., Foster City,...

1. A method comprising:receiving image data;
inputting at least a portion of the image data into a machine-learning (ML) model;
determining, by the ML model, a region of interest (ROI) associated with an object that appears in the image;
determining, by the ML model and based at least in part on the ROI, additional outputs, the additional outputs comprising:
a semantic segmentation associated with the object, the semantic indicating a classification of the object;
directional data that indicates a center of the object;
depth data associated with at least the portion of the image; and
an instance segmentation associated with the object;
determining a consistency loss based at least in part on two or more of the ROI, the semantic segmentation, the directional data, the depth data, or the instance segmentation;
altering, as a trained ML model and based at least in part on the consistency loss, one or more parameters of the ML model; and
transmitting the trained ML model to an autonomous vehicle.

US Pat. No. 10,929,711

TIME OF FLIGHT DATA SEGMENTATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving sensor data of an environment captured by a time of flight sensor on an autonomous vehicle, the sensor data arranged as a two dimensional image, wherein an individual element of one or more channels of the two dimensional image comprises one or more of depth information, intensity information, or confidence information;
determining, based at least in part on the two dimensional image, three dimensional data, wherein point data of the three dimensional data represents the sensor data in three dimensions;
associating, as voxel data, the three dimensional data with a voxel space;
receiving ground plane data associated with a location of the autonomous vehicle;
determining, based at least in part on the ground plane data, that a portion of the voxel data represents a ground surface;
removing the ground surface from the voxel data to determine a subset of the voxel data;
clustering voxels in the subset of the voxel data to determine an object;
associating the object with at least one of a dynamic grid or a static grid; and
controlling the autonomous vehicle based at least in part on the object associated with at least one of the dynamic grid or the static grid.

US Pat. No. 10,921,817

POINT CLOUD FILTERING WITH SEMANTIC SEGMENTATION

Zoox, Inc., Foster City,...

1. An autonomous vehicle comprising:one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
receiving, from a LIDAR device on the autonomous vehicle, LIDAR data associated with an environment proximate the autonomous vehicle, the LIDAR data comprising a set of LIDAR points;
receiving, from a camera on the autonomous vehicle, image data associated with an image of a portion of the environment;
determining that the set of LIDAR points represents a non-navigable space;
projecting, as a LIDAR projection onto the image, a voxel associated with a point of the set of LIDAR points;
determining, based at least in part on a label associated with the image, a classification for a pixel associated with the voxel; and
determining, based at least in part on the classification, that the set of LIDAR points represents a navigable space for the autonomous vehicle.

US Pat. No. 10,884,428

MESH DECIMATION TECHNIQUES AND VALIDATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the system to:
receive first sensor data including at least LIDAR data representing an environment;
generate, based at least in part on the first sensor data, a three-dimensional map of the environment, the three-dimensional map including at least a mesh representing the environment with a first plurality of polygons;
receive second sensor data captured by sensors disposed on a vehicle traversing the environment;
determine, based at least in part on the second sensor data and the mesh, a first pose of the vehicle in the environment;
determine, based at least in part on the first pose, a first accuracy level associated with the mesh;
generate a decimated mesh by applying at least one decimation algorithm to the mesh, the decimated mesh representing the environment with a second plurality of polygons;
determine, based at least in part on the second sensor data and the decimated mesh, a second pose of the vehicle in the environment;
determine, based at least in part on the second pose, a second accuracy level associated with the decimated mesh;
determine that the second accuracy level is within a threshold value of the first accuracy level; and
send at least a portion of the decimated mesh to an autonomous vehicle for controlling the autonomous vehicle to navigate within the environment.

US Pat. No. 10,831,214

MESH DECIMATION TECHNIQUES AND VALIDATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the system to:
receive first sensor data including at least LIDAR data representing an environment;
generate, based at least in part on the first sensor data, a three-dimensional map of the environment, the three-dimensional map including at least a mesh representing the environment with a first plurality of polygons;
receive second sensor data captured by sensors disposed on a vehicle traversing the environment;
determine, based at least in part on the second sensor data and the mesh, a first pose of the vehicle in the environment;
determine, based at least in part on the first pose, a first accuracy level associated with the mesh;
generate a decimated mesh by applying at least one decimation algorithm to the mesh, the decimated mesh representing the environment with a second plurality of polygons;
determine, based at least in part on the second sensor data and the decimated mesh, a second pose of the vehicle in the environment;
determine, based at least in part on the second pose, a second accuracy level associated with the decimated mesh;
determine that the second accuracy level is within a threshold value of the first accuracy level; and
send at least a portion of the decimated mesh to an autonomous vehicle for controlling the autonomous vehicle to navigate within the environment.

US Pat. No. 10,832,502

CALIBRATION FOR AUTONOMOUS VEHICLE OPERATION

Zoox, Inc., Foster City,...

15. One or more non-transitory computer readable media having instructions stored thereon which, when executed by one or more processors of a system, cause the system to perform operations comprising:receiving first sensor data associated with a first measurement made by a first sensor disposed about the system, the first sensor comprising an image sensor;
receiving second sensor data associated with a second sensor disposed about the system, the second sensor comprising a LIDAR sensor;
determining, based at least in part on the first sensor data and the second sensor data, that the first sensor data is anomalous, wherein determining that the first sensor data is anomalous comprises:
detecting a set of detected edges in the second data;
determining, as region data, at least a portion of the first sensor data with the set of detected edges; and
determining, based at least in part on the region data, that the first sensor data is anomalous; and
determining an expected sensor measurement based at least in part on the first measurement and the second sensor data;
determining a calibration parameter associated with the first sensor based at least in part on the expected sensor measurement; and
calibrating the first sensor based at least in part on the calibration parameter.

US Pat. No. 10,817,740

INSTANCE SEGMENTATION INFERRED FROM MACHINE LEARNING MODEL OUTPUT

Zoox, Inc., Foster City,...

1. A method comprising:providing, as input to a machine-learning (ML) model, an image;
receiving, as output from the ML model, a feature map comprising a plurality of features, the plurality of features comprising a plurality of regions of interest (ROIs) and an individual feature of the plurality of features associated with a portion of the image and comprising a confidence score and a first region of interest (ROI);
determining, based at least in part on a non-maximal suppression (NMS) technique, an output ROI, the NMS technique comprising determining a subset of the plurality of ROIs; and
determining an instance segmentation based at least in part on determining portions of the image associated with the subset, the instance segmentation identifying a portion of the image as being associated with an object.

US Pat. No. 10,705,194

AUTOMATED DETECTION OF SENSOR MISCALIBRATION

Zoox, Inc., Foster City,...

1. A method for detecting a miscalibrated sensor, the method comprising:capturing sensor data of an environment using a plurality of sensors disposed about an autonomous vehicle, the sensor data comprising first light detecting and ranging (LIDAR) data associated with a first LIDAR sensor, second LIDAR data associated with a second LIDAR sensor, and third LIDAR data associated with a third LIDAR sensor;
determining a region associated with a ground surface of the environment;
determining, as region data, a subset of data of the sensor data associated with the region;
determining that the region data corresponds to a flat surface;
determining a first average height of a first subset of the region data associated with a first portion of the region, the first subset comprising first data associated with the first LIDAR data;
determining a second average height of a second subset of the region data associated with a second portion of the region, wherein at least a part of the first portion is different than the second portion, the second subset comprising second data associated with the second LIDAR data and the third LIDAR data;
determining a difference between the first average height and the second average height;
determining that the difference meets or exceeds a threshold height value; and
generating an indication that a sensor is miscalibrated.

US Pat. No. 10,584,971

VERIFICATION AND UPDATING OF MAP DATA

Zoox, Inc., Foster City,...

1. A computer-implemented method comprising:receiving, at a global map server, first local map data from a first electronic device, the first local map data including at least one of first location data or first image data for a first area included in a global map;
comparing, with the global map server, the first local map data to a portion of the global map, the portion of the global map related to the first area;
identifying, with the global map server, a difference between the first local map data and the portion of the global map;
sending, from the global map server to one or more additional electronic devices, a mapping request for the first area based on the identification of the difference;
receiving, at the global map server, second local map data from a second electronic device from among the one or more additional electronic devices, the second local map data including at least one of second location data or second image data for the first area;
comparing, with the global map server, the second local map data to the first local map data, the portion of the global map, or both the first local map data and the portion of the global map; and
confirming the difference between the first local map data and the portion of the global map.

US Pat. No. 10,582,137

MULTI-SENSOR DATA CAPTURE SYNCHRONIZAITON

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
determining a first pose of an image sensor, wherein the image sensor is associated with a first field of view;
determining a second pose of a LIDAR sensor, wherein the LIDAR sensor is associated with a second field of view; and
causing the image sensor to initiate a rolling shutter image capture of the first field of view, such that at least a first portion of the first field of view overlaps at least a second portion of the second field of view in accordance with one or more synchronization conditions, wherein the rolling shutter image capture produces an image frame that comprises:
a first scan line associated with a first exposure time; and
a second scan line associated with a second exposure time that is different than the first exposure time.

US Pat. No. 10,423,934

AUTOMATED VEHICLE DIAGNOSTICS AND MAINTENANCE

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more non-transitory computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
receive at least one of sensor data or one or more error codes associated with an autonomous vehicle;
determine at least one servicing issue associated with the autonomous vehicle based at least in part on the at least one of the sensor data or the one or more error codes, the at least one servicing issue associated with a cleanliness of the autonomous vehicle, a potential hardware malfunction associated with the autonomous vehicle, or a state of the autonomous vehicle;
determine that the autonomous vehicle is to be serviced by a mobile technician based at least in part on the at least one servicing issue associated with the autonomous vehicle;
determine a technician skill level associated with the at least one servicing issue, the technician skill level indicating a qualification of technicians to resolve the at least one servicing issue;
select a technician from one or more technicians to service the at least one servicing issue associated with the autonomous vehicle, wherein the technician is selected based at least in part on i) the technician skill level, ii) a distance between a first location of the autonomous vehicle and a second location of the technician, and iii) a traffic level associated with the first location or the second location; and
provide instructions to the autonomous vehicle to navigate to the second location of the technician.

US Pat. No. 10,409,278

MACHINE-LEARNING SYSTEMS AND TECHNIQUES TO OPTIMIZE TELEOPERATION AND/OR PLANNER DECISIONS

Zoox, Inc., Foster City,...

1. A method comprising:receiving telemetry data from an autonomous vehicle, the telemetry data comprising sensor data from a sensor on the autonomous vehicle;
receiving policy data;
determining, based at least in part on at least one of the telemetry data or the policy data, an event in a region of an environment through which the autonomous vehicle has traversed;
determining, based at least in part on the event, that a state of operation of the autonomous vehicle is non-normative, the non-normative state indicating an insufficiency of guaranteeing collision-free travel;
determining, based at least in part on the state of operation, a plurality of trajectories;
determining, based at least in part on a trajectory of the plurality of trajectories, updated policy data; and
transmitting the updated policy data to the autonomous vehicle, the updated policy data configured to cause the autonomous vehicle to operate according to one or more policies of the updated policy data.

US Pat. No. 11,062,454

MULTI-MODAL SENSOR DATA ASSOCIATION ARCHITECTURE

Zoox, Inc., Foster City,...


1. A method comprising:receiving first sensor data associated with a first type of sensor, the first sensor data representing a portion of an environment surrounding an autonomous vehicle;
receiving second sensor data associated with a second type of sensor, the second sensor data representing a same portion or different portion of the environment as the portion represented by the first sensor data;
receiving an object detection, wherein the object detection identifies an object in one or more images;
determining, based at least in part on the object detection, a first subset of the first sensor data and a second subset of the second sensor data;
inputting the first subset of the first sensor data into a first subnetwork;
inputting the second subset of the second sensor data into a second subnetwork;
receiving a first output from the first subnetwork and a second output from the second subnetwork;
combining, as a combined output, the first output and the second output;
inputting a first portion of the combined output into a third subnetwork and a second portion of the combined output into a fourth subnetwork;
receiving a first map from the third subnetwork and a second map from the fourth subnetwork, wherein:the first map indicates at least a first probability that a first point of the first sensor data is associated with the object, and
the second map indicates at least a second probability that a second point of the second sensor data is associated with the object; and

determining, based at least in part on at least one of the first map or the second map, a three-dimensional region of interest.

US Pat. No. 10,960,776

REDUNDANT BATTERY MANAGEMENT SYSTEM ARCHITECTURE

Zoox, Inc., Foster City,...

1. A system comprising:a first battery unit comprising a first battery and a first battery management system (BMS);
a second battery unit comprising a second battery and a second BMS; and
a controller communicatively coupled with the first BMS and the second BMS, wherein the controller is configured to perform operations comprising:
receiving first battery data and first aggregate limit data from the first BMS, wherein the first battery data comprises a first current limit or a first power limit associated with the first battery;
receiving second battery data and second aggregate limit data from the second BMS, wherein the second battery data comprises a second current limit or a second power limit associated with the second battery;
determining, based at least in part on the first aggregate limit data and the second aggregate limit data, an operational limit;
determining, based at least in part on the operational limit, a torque value of an electric motor; and
controlling the electric motor based at least in part on the torque value.

US Pat. No. 10,955,851

DETECTING BLOCKING OBJECTS

Zoox, Inc., Foster City,...

1. An autonomous vehicle comprising:at least one sensor;
a drive system to control physical operation of the autonomous vehicle; and
a perception engine configured to perform operations comprising:
receiving sensor data from the at least one sensor;
detecting, by a first machine-learning model and based at least in part on the sensor data, a stationary vehicle in an environment of the autonomous vehicle;
determining, based at least in part on the sensor data, one or more feature values including an indication of at least one of a presence of one or more other vehicles, one or more other vehicle speeds, or a distribution of traffic flow;
providing, as input to a second machine-learning model, the one or more feature values;
receiving, from the second machine-learning model, an indication that the stationary vehicle is a blocking vehicle or a non-blocking vehicle; and
transmitting the indication to a planner, wherein the planner is configured to perform operations comprising:
receiving the indication; and
generating a trajectory for controlling motion of the autonomous vehicle.

US Pat. No. 10,948,147

MULTIFUNCTION LIGHTING UNIT

Zoox, Inc., Foster City,...

1. A method comprising:receiving, based at least in part on a vehicle travelling in a first direction, a first signal to cause a first light source to emit light representing a headlight of the vehicle; and
receiving, based at least in part on the vehicle travelling in a second direction opposite the first direction, at least one of:
a second signal to cause a second light source to emit light representing a tail light of the vehicle; or
a third signal to cause a third light source to emit light representing a brake light of the vehicle;
wherein the second light source and the third light source are associated with a first structure that is adjacent to a second structure, wherein the second structure comprises first optics positioned to be illuminated by the second light source and second optics positioned to be illuminated by the third light source.

US Pat. No. 10,922,574

BOUNDING BOX EMBEDDING FOR OBJECT IDENTIFYING

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
training a machine learned (ML) model based at least in part on a subset of examples, the training comprising:
identifying a reference region associated with an object represented in an image;
determining a training bounding box associated with the object;
determining first embeddings for pixels of the reference region;
associating a second embedding for the training bounding box with the first embeddings; and
minimizing a loss function with respect to a difference between the first embeddings associated with the object and the second embedding associated with the training bounding box; and
transmitting the ML model to an autonomous vehicle for detecting one or more additional objects in an environment.

US Pat. No. 10,921,811

ADAPTIVE AUTONOMOUS VEHICLE PLANNER LOGIC

Zoox, Inc., Foster City,...

1. A method comprising:receiving path data with which to guide motion of an autonomous vehicle from a first geographic location to a second geographic location;
generating, based, at least in part, on the path data, a trajectory;
generating, based, at least in part, on the path data, a contingent trajectory;
controlling the autonomous vehicle according to the trajectory; and
controlling the autonomous vehicle according to the contingent trajectory subsequent to an impairment of the trajectory.

US Pat. No. 10,916,035

CAMERA CALIBRATION USING DENSE DEPTH MAPS

Zoox, Inc., Foster City,...

1. A system comprising:a vehicle configured to traverse through an environment;
a sensor array on the vehicle, the sensor array comprising a first camera having a first field of view, a second camera having a second field of view at least partially overlapping the first field of view, and a LiDAR sensor having a third field of view at least partially overlapping the first field of view;
one or more processors; and
computer-readable media storing instructions executable by the one or more processors, wherein the instructions cause the system to perform operations comprising:
receiving, from the LiDAR sensor, point cloud data;
generating, based at least in part on the point cloud data, a depth map in a two-dimensional space;
receiving, from the first camera, a first image;
receiving, from the second camera, a second image;
transferring, based at least in part on the depth map, at least a portion of the first image corresponding to a portion of the first field of view overlapping the second field of view into an image plane associated with the second camera as a first projected image;
determining a difference between a feature in the first projected image and the feature in the second image;
generating, based at least in part on the difference, a calibration function for calibrating at least one of the first camera relative to the second camera or the second camera relative to the first camera; and
calibrating, based at least in part on the calibration function, additional images captured by at least one of the first camera or the second camera as calibrated data.

US Pat. No. 10,837,788

TECHNIQUES FOR IDENTIFYING VEHICLES AND PERSONS

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving, from a user device, a request for a vehicle to navigate to a pickup location to pick up a passenger;
commanding the vehicle to navigate towards the pickup location;
acquiring image data from an image sensor on the vehicle, the image data comprising data representing a person proximate to the pickup location;
sending at least a portion of the image data representing the person to the user device;
receiving, from the user device, a confirmation that the person is the passenger; and
commanding, based at least in part on the confirmation, the vehicle to commence a navigation with the person as the passenger.

US Pat. No. 10,831,210

TRAJECTORY GENERATION AND OPTIMIZATION USING CLOSED-FORM NUMERICAL INTEGRATION IN ROUTE-RELATIVE COORDINATES

Zoox, Inc., Foster City,...

1. A system comprising:a processor; and
a computer-readable media storing instructions executable by the processor, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving a reference trajectory comprising a plurality of target velocities and a plurality of target curvatures, wherein a target velocity of the plurality of target velocities is associated with a corresponding target curvature of the plurality of target curvatures;
optimizing a desired trajectory with respect to the reference trajectory to generate an optimized trajectory comprising a plurality of optimized target velocities and a plurality of optimized target curvatures, wherein:
an optimized target velocity of the plurality of optimized target velocities is associated with a corresponding optimized target curvature of the plurality of optimized target curvatures;
an acceleration associated with the plurality of target velocities and the plurality of target curvatures are optimized substantially simultaneously; and
the optimizing is based at least in part on a closed-form solution of longitudinal dynamics of a vehicle; and
controlling an autonomous vehicle to traverse an environment based at least in part on the optimized trajectory.

US Pat. No. 10,832,414

SENSOR DATA SEGMENTATION

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions executable by the one or more processors to:
receive image data;
store, based at least in part on the image data, a segmented image;
receive sensor data from at least one of a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, or a sound navigation and ranging (SONAR) sensor;
associate the sensor data with the segmented image as part of a training dataset; and
train a sensor data segmentation model using the training dataset such that, upon receiving input comprising at least one of additional LIDAR sensor data, additional RADAR sensor data, or additional SONAR sensor data, the sensor data segmentation model is configured to output at least one of segmented additional LIDAR sensor data, segmented additional RADAR sensor data, or segmented additional SONAR sensor data.

US Pat. No. 11,079,492

CONDITION DEPENDENT PARAMETERS FOR LARGE-SCALE LOCALIZATION AND/OR MAPPING

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:receiving LIDAR data from a LIDAR sensor of an autonomous vehicle;
determining a location of the autonomous vehicle within a map based at least in part on the LIDAR data;
selecting, based at least in part on the location, a parameter associated with a configuration of a mapping component that creates or updates the map based at least in part on the LIDAR data; and
updating the map, based at least in part on additional LIDAR data and the parameter, of an environment through which the autonomous vehicle traverses.


US Pat. No. 11,048,260

ADAPTIVE SCALING IN TRAJECTORY GENERATION

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:receiving a reference trajectory representing an initial trajectory for an autonomous vehicle to traverse an environment;
determining at least one of:a first cost associated with a curvature value associated with a first point of the reference trajectory; or
a second cost associated with a distance between the first point and a point associated with an obstacle in the environment;

determining, based at least in part on at least one of the first cost or the second cost, a first point density of first points associated with a first portion of the reference trajectory;
determining a second point density of second points associated with a second portion of the reference trajectory, wherein the second point density is less than the first point density;
evaluating a cost function at the first point of the first points and at a second point of the second points to generate a target trajectory comprising first corresponding points and second corresponding points, wherein a first corresponding point of the first corresponding points corresponds to the first point of the first points, and wherein a second corresponding point of the second corresponding points corresponds to a second point of the second points; and
controlling the autonomous vehicle to traverse the environment based at least in part on the target trajectory.


US Pat. No. 11,048,265

OCCLUSION AWARE PLANNING

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:capturing LIDAR data using a LIDAR sensor on an autonomous vehicle;
accessing map data representing an environment proximate to the autonomous vehicle, the map data comprising an occlusion grid comprising multiple occlusion fields, wherein an occlusion field of the multiple occlusion fields is associated with an occlusion state and an occupancy state of a portion of the environment;
ray casting a portion of the LIDAR data to determine the occupancy state and the occlusion state of the occlusion field;
determining, based at least in part on the occlusion field, an amount of information associated with the occlusion grid to determine a safe trajectory for the autonomous vehicle to traverse the environment, wherein the amount of information comprises one or more of:a first number of occlusion fields associated with an unoccupied occupancy state;
a second number of observations associated with the occlusion field and a period of time;
a third number of occlusion fields associated with a visible occlusion state; or
a size of an unoccupied and un-occluded portion of the environment;

determining that the amount of information meets or exceeds a threshold level; and
controlling the autonomous vehicle to traverse the environment in accordance with the safe trajectory.


US Pat. No. 10,984,543

IMAGE-BASED DEPTH DATA AND RELATIVE DEPTH DATA

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
capturing, by an image sensor of an autonomous vehicle, image data associated with an environment;
determining an object represented in the image data;
capturing, by a lidar sensor of the autonomous vehicle, lidar data associated with the object, a portion of the lidar data associated with the object comprising a reference point;
inputting a crop of the image data and the reference point to a machine-learned model;
receiving, from the machine-learned model, relative depth data associated with the object, the relative depth data comprising depth data that is relative to a depth associated with the reference point; and
determining, based at least in part on the reference point and the relative depth data, estimated depth data.

US Pat. No. 10,976,732

PREDICTIVE TELEOPERATOR SITUATIONAL AWARENESS

Zoox, Inc., Foster City,...

1. A method comprising:obtaining data from a vehicle;
determining a representation of data based at least in part on a subset of the data, wherein the subset comprises at least one of a request associated with the vehicle, sensor data, a detected object, event data, vehicle state data, or environmental data;
providing, as input to a machine-learned model, the representation of data;
receiving, from the machine-learned model and based at least in part on the representation of data, at least one of a teleoperator option or a presentation configuration;
causing at least one of presentation of the teleoperator option via a user interface or configuration of the user interface based at least in part on the presentation configuration; and
modifying the machine-learned model, wherein modifying the machine-learned model comprises adjusting a parameter of the machine-learned model to decrease a time between receiving the data from the vehicle and transmitting a guidance signal to the vehicle.

US Pat. No. 10,963,709

HIERARCHICAL MACHINE-LEARNING NETWORK ARCHITECTURE

Zoox, Inc., Foster City,...

1. A method comprising:receiving, from a sensor of an autonomous vehicle, an image;
providing, as input to a first neural network, the image;
receiving, from the first neural network, a feature map, a region of interest, a classification, and a first probability associated with an object represented in the image;
providing, as input to a second neural network, at least a portion of the feature map that corresponds to the region of interest;
receiving, from the second neural network, a sub-classification of the classification and a second probability associated with the subclassification; and
controlling operation of the autonomous vehicle based at least in part on at least one of the classification or the sub-classification.

US Pat. No. 10,960,939

WORM GEAR DRIVE UNIT INTERFACE AND ASSEMBLY METHODS

Zoox, Inc., Foster City,...

1. A system comprising:a drive unit coupleable to a body of a vehicle using a drive system comprising:
a first driveshaft, with a first end and a second end;
a first worm shaft coupled to the first driveshaft;
a first worm gear in toothed engagement with the first worm shaft;
a first driven gear, concentrically disposed inside the first worm gear and in toothed engagement with the first worm gear, the first driven gear to rotate a first fastener;
a first drive housing to at least partially enclose the first worm shaft, first worm gear, and first driven gear;
a first removable cover coupled to the first drive housing;
a second driveshaft, with a third end and a fourth end, at least partially disposed concentrically within the first driveshaft;
a second worm shaft coupled to the second driveshaft;
a second worm gear in toothed engagement with the second worm shaft; and
a second driven gear, concentrically disposed inside the second worm gear and in toothed engagement with the second worm gear, the second driven gear to rotate a second fastener;
a second drive housing to at least partially enclose the second worm shaft, second worm gear, and second driven gear; and
a second removable cover coupled to the second drive housing;
wherein rotating the first driveshaft causes the first driven gear to rotate the first fastener to couple the drive unit to the body; and
wherein rotating the second driveshaft causes the second driven gear to rotate the second fastener to further couple the drive unit to the body.

US Pat. No. 10,960,744

MULTI-DIRECTIONAL VEHICLE ROOF COVER SYSTEM

Zoox, Inc., Foster City,...

1. A vehicle comprising:a body comprising a compartment having a first end, a second end opposite the first end, and a top, the compartment having an opening in the top;
a cover movably coupled to the body to selectively cover the opening in the compartment, the cover having:
a first end proximate the first end of the compartment;
a second end proximate the second end of the compartment;
lateral edges connecting the first end and the second end; and
linkages coupling the lateral edges to the body;
the cover being actuatable between:
a first position in which the cover is closed and covers the opening in the compartment;
a second position in which the cover is open and the first end of the cover is raised relative to the second end of the cover; and
a third position in which the cover is open and the second end of the cover is raised relative to the first end of the cover; and
one or more processors and memory, the memory storing instructions that, when executed by the one or more processors, configure the vehicle to perform operations comprising:
controlling the vehicle to operate with the cover in one of the second position or the third position;
detecting a condition associated with the vehicle; and
actuating, based at least in part on detecting the condition, the cover to move from the one of the second position or the third position to the first position.

US Pat. No. 10,875,435

HEADREST WITH PASSENGER FLAPS

Zoox, Inc., Foster City,...

1. A system comprising:one or more processors;
a headrest;
a flap pivotably coupled to the headrest;
an actuator to move the flap; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions that are executable by the one or more processors to:
determine a passenger state of a seat in an autonomous vehicle;
determine a first flap state of the flap relative to the headrest in the autonomous vehicle; and
cause the actuator to engage the flap based, at least in part, on one or more of the passenger state and the first flap state, to move the flap relative to the headrest from the first flap state to a second flap state different than the first flap state,
wherein the flap is pivotable along a pivot line that extends vertically a height of the headrest.

US Pat. No. 10,832,439

LOCATING ENTITIES IN A MAPPED ENVIRONMENT

Zoox, Inc., Foster City,...

1. A vehicle comprising:at least one sensor configured to generate sensor data of an environment of the vehicle;
one or more processors; and
memory storing processor-executable instructions that, when executed by the one or more processors, cause the autonomous vehicle to perform acts comprising:
determining, based at least in part on the sensor data, a location of an entity in the environment, the location comprising a coordinate in a global coordinate system;
determining, based at least in part on an axis-aligned bounding box (AABB) tree and the location, a road segment associated with the location, the AABB tree describing relationships between a plurality of AABBs representative of a plurality of road segments in the environment;
determining, based at least in part on the road segment, a segment-centric location of the entity in a segment-centric coordinate system;
generating, based at least in part on the segment-centric location of the entity in the road segment, one or more controls to control the vehicle relative to the entity; and
controlling the vehicle according to the one or more controls.

US Pat. No. 10,831,188

REDUNDANT POSE GENERATION SYSTEM

Zoox, Inc., Foster City,...

1. A system embedded in a vehicle, the system comprising:one or more sensors configured to generate sensor data;
a first pose estimation component configured to determine, based at least in part on at least a first portion of the sensor data, a first pose of the vehicle;
a second pose estimation component configured to:
determine, based at least in part on at least a second portion of the sensor data, a second pose of the vehicle; and
determine a difference between the first pose and the second pose, wherein determining the difference between the first pose and the second pose comprises:
determining one or more of a distance between the first pose and the second pose or a difference in an orientation of the first pose and the second pose; and
determining that the distance meets or exceeds a location threshold or the difference in the orientation meets or exceeds an orientation threshold;
one or more components configured to generate, based at least in part on the difference between the first pose and the second pose, a trajectory for the vehicle; and
one or more components configured to cause the vehicle to follow the trajectory.

US Pat. No. 10,780,930

WORM GEAR DRIVE UNIT INTERFACE AND ASSEMBLY METHODS

Zoox, Inc., Foster City,...

1. A robotic cart comprising:a drive system to move the robotic cart throughout an area;
one or more transceivers to send and receive one or more wired and wireless transmissions;memory storing at least a localization module and torque data;one or more inputs to receive data from a user; andone or more processors in communication with at least the one or more transceivers,
the memory, and
the one or more inputs,
the memory including
computer executable instructions to
cause the one or more processors to:
receive sensor data from
one or more sensors disposed about the robotic cart;
determine, using the sensor data, a pose of the robotic cart relative to a vehicle;
plan a trajectory of the robotic cart based, at least in part, on the pose,
the trajectory causing the robotic cart to align with one or more couplers of the vehicle;
cause a drive unit of a robotic cart to move the robotic cart along the trajectory;
send a first signal to torquing device on the robotic cart,
the first signal causing a torquing device to rotate a drive interface in a first direction to remove a fastener from the vehicle, the fastener detachably coupling a subassembly to the vehicle, the subassembly comprising a portion of the vehicle; and
move, using the drive unit, the robotic cart in a second direction away from
the vehicle to remove the subassembly from the vehicle.

US Pat. No. 10,692,377

ENHANCED TRAVEL MODES FOR VEHICLES

Zoox, Inc., Foster City,...

1. A method, comprising:receiving, at a computing device, signals from a plurality of vehicles, the signals comprising information indicative of a current location of one or more vehicles of the plurality of vehicles;
receiving, at the computing device and from a first vehicle of the plurality of vehicles, a request for preferential travel from a current location of the first vehicle to a particular destination, wherein
the first vehicle is permitted to operate in accordance with a first set of parameters, and
a second vehicle of the plurality of vehicles is permitted to operate in accordance with a second set of parameters;
determining, at the computing device and based at least partly on the request and the signals,
a third set of parameters, the third set of parameters differing from the first set of parameters, and
a fourth set of parameters, the fourth set of parameters requiring the second vehicle to remain outside of a drive envelope of the first vehicle as the first vehicle travels to the particular destination;
providing a first command signal to the first vehicle with the computing device, the first command signal including the third set of parameters; and
providing a second command signal to the second vehicle with the computing device, the second command signal including the fourth set of parameters.

US Pat. No. 10,614,717

DRIVE ENVELOPE DETERMINATION

Zoox, Inc., Foster City,...

1. A vehicle comprising:one or more sensors disposed on the vehicle;
one or more processors; and
memory storing processor-executable instructions that, when executed by the one or more processors, configure the vehicle to:
receive sensor data from a sensor of the one or more sensors, the sensor data comprising information about an object in an environment proximate the vehicle;
determine a planned path of the vehicle through the environment;
determine a lateral distance between the planned path and the object;
determine a classification of the object;
determine, based at least in part on the lateral distance and the classification, a width; and
determine a drive envelope, wherein the drive envelope defines a boundary of a portion of the environment along the planned path in which the vehicle can travel and wherein at least a portion of the envelope extends the width laterally from the planned path and toward the object.

US Pat. No. 11,126,873

VEHICLE LIGHTING STATE DETERMINATION

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:capturing image data of an environment using an image sensor on an autonomous vehicle, the image data comprising a plurality of frames captured over time;
identifying a vehicle in the plurality of the frames as a tracked object in the environment;
extracting, from a first frame of the image data, first cropped image data representing the tracked object at a first time;
extracting, from a second frame of the image data, second cropped image data representing the tracked object at a second time;
inputting, substantially simultaneously, a subset of the image data into an input layer of a machine learning algorithm to cause the machine learning algorithm to perform substantially simultaneous analysis on at least the first cropped image data and the second cropped image data, the inputting, substantially simultaneously, the subset of the image data into the input layer of the machine learning algorithm comprising:inputting, substantially simultaneously, at least the first cropped image data into a first input of the input layer of the machine learning algorithm and inputting the second cropped image data into a second input of the input layer of the machine learning algorithm;


receiving, from the machine learning algorithm, a probability that the first cropped image data representing the tracked object is associated with a lighting state of the tracked object, wherein the lighting state comprises a left blinker state, a right blinker state, a braking state, and a hazard state and the probability comprises:a first probability associated with the left blinker state of the tracked object;
a second probability associated with the right blinker state of the tracked object;
a third probability associated with the braking state of the tracked object; and
a fourth probability associated with the hazard state of the tracked object;

generating, based at least in part on the probability, a trajectory for the autonomous vehicle; and
causing the autonomous vehicle to follow the trajectory.

US Pat. No. 11,125,861

MESH VALIDATION

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:receiving LIDAR data representing an environment;
associating the LIDAR data with a voxel space, the voxel space comprising a plurality of voxels;
identifying a first voxel and a second voxel from the plurality of voxels;
analyzing first voxel data associated with the first voxel and second voxel data associated with the second voxel to identify an error in at least one of the first voxel data or the second voxel data, the error indicating at least one of:two or more ground surfaces located less than a first threshold distance from each other;
two or more wall surfaces located less than a second threshold distance from each other;
a hole in a ground or wall surface; or
an angular difference of surface normal vectors that meets or exceeds a threshold value;

generating a mesh based at least in part on the first voxel data and the second voxel data; and
performing an action based at least in part on the error, the action comprising at least one of:causing display of a representation of the mesh with an indication of the error;
storing an indication of the error in a log file;
causing a trajectory for an autonomous vehicle to be updated; or
automatically updating the mesh.



US Pat. No. 11,126,178

VEHICLE CONTROL

Zoox, Inc., Foster City,...


1. An autonomous vehicle comprising:a first controller for determining first accelerations based at least in part on first input information and a first controller gain;
a second controller for determining second accelerations based at least in part on second input information and a second controller gain;
one or more processors; and
memory storing processor-executable instructions that, when executed by the one or more processors, cause the autonomous vehicle to perform acts comprising:receiving a first request for controlling the autonomous vehicle, the first request comprising first velocity information and distance information;
determining, using the first controller, a first acceleration based at least in part on the first request;
receiving a second request for controlling the autonomous vehicle, the second request comprising second velocity information;
determining, using the second controller, a second acceleration based at least in part on the second request;
determining, based at least in part on the first acceleration and the second acceleration, a command acceleration for a point along a trajectory for controlling the vehicle; and
controlling the vehicle based at least in part on the trajectory.


US Pat. No. 11,126,179

MOTION PREDICTION BASED ON APPEARANCE

Zoox, Inc., Foster City,...


1. A system for training a machine learned model to detect object speed, the system comprising:one or more processors; and
one or more non-transitory computer-readable media storing instructions that when executed by the one or more processors, cause the system to perform operations comprising:receiving first sensor data captured by one or more sensors of an autonomous vehicle at a first time, the first sensor data comprising first image data;
detecting, based at least in part on the first image data, a pedestrian having a pose in an environment of the autonomous vehicle;
determining, based at least in part on one or more of the first sensor data or second sensor data captured by the one or more sensors at a second time comprising a second image associated with a representation of the pedestrian, a measured speed of the pedestrian;
inputting, into a machine learned model, the first image data;
receiving, from the machine learned model, a predicted speed of the pedestrian that is based on the first image data independent of sensor data from other sensor modalities;
determining a difference between the predicted speed of the pedestrian and the measured speed of the pedestrian; and
altering one or more parameters of the machine learned model to minimize the difference to obtain a trained machine learned model to detect object speed.


US Pat. No. 11,126,180

PREDICTING AN OCCUPANCY ASSOCIATED WITH OCCLUDED REGION

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:capturing sensor data of an environment using a sensor of an autonomous vehicle;
determining, based at least in part on the sensor data, a first location of an object in the environment;
receiving a candidate trajectory for the autonomous vehicle to follow;
determining a second location of the autonomous vehicle along the candidate trajectory;
determining, based at least in part on the sensor data, the first location, and the second location, an occluded region in the environment occluded by the object;
representing the environment as top-down data comprising the occluded region;
inputting the top-down data into a machine learned model;
receiving, from the machine learned model, prediction probabilities associated with an occupancy of the occluded region;
determining, based at least in part on the prediction probabilities and the candidate trajectory, a cost associated with the candidate trajectory;
determining that the cost is below a cost threshold; and
controlling the autonomous vehicle to follow the candidate trajectory based at least in part on the cost being below the cost threshold.


US Pat. No. 11,124,154

TECHNIQUES FOR AUTHORIZING VEHICLES

Zoox, Inc., Foster City,...


1. A system comprising:one or more network components;
one or more processors; and
one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:receiving, from an autonomous vehicle, a reported location;
sending, to the autonomous vehicle, a request for sensor data;
receiving, from the autonomous vehicle, sensor data representing a portion of an environment of the autonomous vehicle;
determining, based at least in part on the sensor data and at least a portion of map data, a sensor-based location of the autonomous vehicle;
determining a difference between the reported location and the sensor-based location;
determining that the difference meets or exceeds a threshold; and
based at least in part on the difference meeting or exceeding the threshold, determining that the autonomous vehicle is not located at the reported location.


US Pat. No. 11,124,185

PERCEPTION COLLISION AVOIDANCE

Zoox, Inc., Foster City,...


1. A vehicle system comprising:one or more sensors;
a first system comprising one or more first processors and one or more first memories comprising instructions that, when executed by the one or more first processors, cause the first processors to:receive first sensor data from a first subset of the one or more sensors; and
determine, based at least in part on the first sensor data, a trajectory for controlling an autonomous vehicle; and

a second system comprising one or more second processors and one or more second memories comprising instructions that, when executed by the one or more second processors, cause the one or more second processors to:receive the trajectory from the first system;
receive second sensor data from a second subset of the one or more sensors;
based at least in part on the trajectory and the second sensor data, determine a probability that the autonomous vehicle will collide with an object;
determine that at least one of an error occurred with the first system or the probability meets or exceeds a probability threshold; and
cause the autonomous vehicle to decelerate along the trajectory based at least in part on determining that the at least one of the error occurred with the first system or the probability meets or exceeds the probability threshold.


US Pat. No. 11,117,543

OCCUPANT PROTECTION SYSTEM INCLUDING INFLATORS

Zoox, Inc., Foster City,...


1. An occupant protection system for a vehicle, the occupant protection system comprising:an expandable curtain configured to expand from a stowed state to a deployed state, wherein the expandable curtain in the deployed state comprises:a first side configured to extend along a portion of a first interior side of the vehicle in a first direction; and
a transverse portion extending in a second direction transverse to the first direction, the transverse portion providing a reaction surface, wherein the first side and the transverse portion of the expandable curtain form a contiguous barrier;

an expandable bladder comprising an occupant facing surface and a rear surface opposite the occupant facing surface, the expandable bladder configured to expand from a stowed state to a deployed state, such that in the deployed state the rear surface of the expandable bladder contacts the reaction surface of the transverse portion of the expandable curtain;
a first inflator in flow communication with the expandable curtain and configured, upon activation of the first inflator, to cause the expandable curtain to expand from the stowed state to the deployed state, to separate at least a portion of a trim panel from a ceiling of the vehicle, and to create an opening though which the expandable curtain and the expandable bladder expand;
a second inflator in flow communication with the expandable bladder and configured, upon activation of the second inflator, to cause the expandable bladder to expand from the stowed state to the deployed state; and
a deployment controller in communication with the first inflator and the second inflator, and configured to activate the first inflator and the second inflator.

US Pat. No. 11,120,538

SENSOR DEGRADATION DETECTION AND REMEDIATION

Zoox, Inc., Foster City,...


1. A vehicle comprising:one or more processors;
a first sensor to capture sensor data of an environment of the vehicle;
a second sensor to capture sensor data of the environment of the vehicle, the second sensor having a field of view that at least partially overlaps with a field of view of the first sensor; and
non-transitory computer-readable media storing instructions that when executed by the one or more processors, cause the vehicle to perform operations comprising:receiving first image data captured by the first sensor at a first time;
identifying a first image region within the first image data;
retrieving second image data captured by the second sensor;
identifying a second image region within the second image data associated with the first image region;
determining a level of visual consistency between the first image region and the second image region;
detecting a degradation of the first image data captured by the first sensor, based on a combination of the determined level of visual consistency and at least one of:an intensity associated with the first image region;
a measurement of temporal movement associated with the first image region, based on third image data captured at a second time different from the first time; or
an output from a machine learned model trained to detect degradations based on image data; and

controlling an operation of the vehicle based at least in part on the detection of the degradation,
wherein detecting the degradation further comprises:inputting, into the machine learned model, the first image data; and
receiving the output from the machine learned model, wherein the output is based at least in part on the first image data, and wherein the output indicates the degradation of the first image data captured by the first sensor.



US Pat. No. 11,110,922

VEHICLE TRAJECTORY MODIFICATION FOR FOLLOWING

Zoox, Inc., Foster City,...


1. A method comprising:determining a drivable area through an environment for a vehicle, wherein the drivable area represents a region in the environment on which the vehicle is permitted to traverse;
determining a trajectory within the drivable area for the vehicle to follow;
receiving sensor data from a sensor on the vehicle;
detecting, based at least in part on the sensor data, an object in the environment;
determining, based at least in part on the sensor data, object parameters associated with the object, wherein the object parameters comprise one or more of a velocity, a position, a classification, or an extent;
determining, as a modified drivable area, based at least in part on the object parameters, and independent of the trajectory, a portion of the drivable area that excludes a space that the object is predicted to occupy based on the object parameters;
determining, based at least in part on the modified drivable area, that the trajectory is invalid;
determining, based at least in part on determining the trajectory is invalid, a modified trajectory; and
controlling the vehicle according to the modified trajectory.

US Pat. No. 11,113,873

MODELING ARTICULATED OBJECTS

Zoox, Inc., Foster City,...


1. A system comprising: one or more processors;non-transitory computer-readable media storing instructions executable by the one or more processors to cause the system to perform operations comprising:
receiving a plurality of points associated with an object;
generating a first model representative of the object, the first model comprising a first rectangle connected to a second rectangle at a pivot, the first rectangle comprising a first end, a second end spaced from the first end by a first length, a first side, and a second side spaced from the first side by a first width, the second rectangle comprising a third end, a fourth end spaced from the third end by a second length, a third side, and a fourth side spaced from the third side by a second width, the pivot being coincident with a first midpoint on the second end and a second midpoint on the third end;
determining, for a point of the plurality of points, a first distance of the point to the first rectangle and a second distance of the point to the second rectangle;
determining, for the point and based at least in part on the first distance and the second distance, a first probability that the point is associated with the first rectangle and a second probability that the point is associated with the second rectangle;
generating, based at least in part on the first probability and the second probability, a second model representative of the object, the second model comprising the first rectangle and the second rectangle, wherein at least one of:the first rectangle is rotated relative to the second rectangle about the pivot, or
at least one of the first width, the second width, the first length, or the second length is altered relative to the first model; and

controlling, based at least in part on the second model as a representation of the object in an environment of an autonomous vehicle, the autonomous vehicle to navigate the environment relative to the object.

US Pat. No. 11,104,394

VEHICLE WITH INTERCHANGEABLE DRIVE MODULES

Zoox, Inc., Foster City,...


1. A drive module for a vehicle comprising:a frame configured to couple a first end of the drive module to a portion of the vehicle;
a propulsion system to propel the vehicle;
a light source configured to emit light into to an environment proximate the vehicle;
a facia coupled to the first end of the drive module, the light source at least one of (i) coupled to the facia, or (ii) disposed in an opening in the facia; and
wherein a second end of the drive module is configured to be a distal end of the vehicle and the light source is disposed proximate the second end of the drive module.

US Pat. No. 11,106,218

ADAPTIVE MAPPING TO NAVIGATE AUTONOMOUS VEHICLES RESPONSIVE TO PHYSICAL ENVIRONMENT CHANGES

Zoox, Inc., Foster City,...


1. A method comprising:receiving, at a map generator, sensor data from one or more autonomous vehicles traveling via a road network, the sensor data being related to an environment through which the one or more autonomous vehicles travel and comprising a first type of sensor data generated by a first type of sensor and a second type of sensor data generated by a second type of sensor different than the first type of sensor;
aligning the first type of sensor data with the second type of sensor data to form aligned subsets of sensor data;
fusing the aligned subsets of sensor data to generate probability distributions associated with one or more environmental properties with respect to one or more locations in an environment;
associating the probability distributions with a map; and
transmitting at least a portion of the map to the one or more autonomous vehicles,
wherein the one or more autonomous vehicles are controlled based at least in part on a probability distribution of the probability distributions associated with the map.

US Pat. No. 11,104,269

DYNAMIC VEHICLE WARNING SIGNAL EMISSION

Zoox, Inc., Foster City,...


1. A vehicle comprising:a sensor;
an emitter;
one or more processors; and
one or more computer-readable media storing instructions that, when executed, configure the vehicle to:determine, based at least in part on sensor data from the sensor, an object in an environment associated with the vehicle;
determine, based at least in part on the sensor data, an object trajectory associated with the object;
determine, based at least in part on the object trajectory, that the object is relevant to a progress of the vehicle;
emit, via the emitter, a first signal based at least in part on determining that the object is relevant to the progress of the vehicle, the first signal comprising a first characteristic;
determine, based at least in part on the sensor data, an object reaction to the first signal;
based at least in part on the object reaction differing from an expected reaction, emit a second signal, wherein the second signal comprises a second characteristic different from the first characteristic; and
store data associated with the first signal and the object reaction in a database.


US Pat. No. 11,104,332

COLLISION AVOIDANCE SYSTEM WITH TRAJECTORY VALIDATION

Zoox, Inc., Foster City,...


1. A vehicle system comprising:a first system comprising one or more first processors and one or more first memories comprising instructions that, when executed by the one or more first processors, cause the one or more first processors to:receive sensor data from one or more sensors; and
generate, based at least in part on the sensor data, a first trajectory for an autonomous vehicle and a second trajectory for the autonomous vehicle; and

a second system comprising one or more second processors and one or more second memories comprising instructions that, when executed by the one or more second processors, cause the one or more second processors to:receive the first trajectory and the second trajectory from the first system;
receive at least a portion of the sensor data;
determine, based at least in part on the at least the portion of the sensor data, an object in an environment;
determine an object trajectory for the object;
perform first collision detection with the first trajectory and the object trajectory to determine a first state associated with the first trajectory, wherein the first collision detection comprises determining whether or not the vehicle would collide with the object based at least in part on the first trajectory and the object trajectory;
perform second collision detection with the second trajectory and the object trajectory to determine a second state associated with the second trajectory, wherein the second collision detection comprises determining whether or not the vehicle would collide with the object based at least in part on the second trajectory and the object trajectory;
determine, based at least in part on the first state and the second state, a third trajectory for the autonomous vehicle, the third trajectory comprising at least one of the first trajectory, the second trajectory, or a fourth trajectory generated by the second system; and
control the autonomous vehicle based at least in part on the third trajectory.


US Pat. No. 11,099,561

CONTROL OF AN AUTONOMOUS VEHICLE IN UNMAPPED REGIONS

Zoox, Inc., Foster City,...


1. A method of controlling a vehicle from a current location in a mapped region to a destination in an unmapped region, comprising:receiving a request to navigate the vehicle to the destination including a structure;
accessing map data comprising information about the mapped region, the vehicle being configured to drive autonomously in the current location of the mapped region using the map data;
determining that the destination is a location in the unmapped region which is bordered by the mapped region;
determining, based at least in part on the destination being the location in the unmapped region, a second location in the mapped region that is proximate and nearer the destination than the current location in the mapped region;
receiving first sensor data from a first sensor disposed on the vehicle and second sensor data from a second sensor disposed on the vehicle;
determining, based at least in part on the first sensor data and the second sensor data, a first control command for causing the vehicle to move to the second location;
based at least in part on the first control command, autonomously controlling a drive system of the vehicle from the current location to the second location within the mapped region;
receiving a second control command for the vehicle to proceed from the second location in the mapped region toward the destination in the unmapped region, the second control command comprising a manual control command to navigate from the second location to the destination;
based at least in part on the second control command, switching from the autonomous control to a manual control of the drive system of the vehicle to move from the second location toward the destination; and
collecting, based at least in part on at least one of the first sensor data or the second sensor data, additional map data about an unmapped region proximate the destination.

US Pat. No. 11,099,563

MULTI-CONTROLLER SYNCHRONIZATION

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:receiving, at a primary computing device, a first trajectory and a second trajectory for an autonomous vehicle, the first trajectory comprising a first set of positions, orientations, and velocities to traverse through an environment and the second trajectory comprising a second set of positions, orientations, and velocities to stop the autonomous vehicle in the environment;
receiving, at a secondary computing device, the second trajectory;
determining, at the primary computing device, vehicle state data, the vehicle state data comprising one or more of a current position, a current velocity, a plurality of previous positions, or a plurality of previous velocities;
determining, at the primary computing device and based at least in part on the vehicle state data and the first trajectory, first internal data;
determining, at the primary computing device and based at least in part on the first internal data, first control data comprising at least one of first steering data or first acceleration data;
transmitting, from the primary computing device to the secondary computing device, one or more of the first internal data or the vehicle state data;
determining, at the secondary computing device, an error state associated with the primary computing device;
determining, at the secondary computing device and based at least in part on the first internal data and the second trajectory, second internal data;
determining, at the secondary computing device and based at least in part on the second internal data, second control data comprising at least one of second steering data or second acceleration data; and
controlling, based at least in part on the second control data and by the secondary computing device, the autonomous vehicle to follow the second trajectory.


US Pat. No. 11,100,339

CLOSED LANE DETECTION

Zoox, Inc., Foster City,...


1. A method comprising:receiving, from one or more sensors, sensor data;
determining, based at least in part on the sensor data, an object detection, the object detection being associated with a lane of a roadway and indicating a location of an object in an environment and a size of the object in at least a first direction along the lane and a second direction perpendicular to the first direction in a plane associated with the lane;
modifying the object detection by increasing the size indicated by the object detection to obtain a dilated object detection;
determining a distance between the dilated object detection and another object detection, another dilated object detection, or an extent of the lane;
determining that the distance is less than or equal to a distance threshold;
determining, based at least in part on the distance being less than or equal to the distance threshold, a closed status indicating that the lane is closed; and
controlling, based at least in part on the closed status, a vehicle.

US Pat. No. 11,099,573

SAFE SYSTEM OPERATION USING LATENCY DETERMINATIONS

Zoox, Inc., Foster City,...


1. An autonomous vehicle comprising:a first sensor configured to generate and output first sensor data;
a second sensor configured to generate and output second sensor data;
one or more processing systems configured to generate and output processed data, the processed data being based at least in part on at least one of the first sensor data or the second sensor data;
a vehicle controller;
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:receiving, at the vehicle controller, a plurality of messages, the vehicle controller being configured to generate and output one or more vehicle control messages based at least in part on the plurality of messages;
determining, based at least in part on the plurality of messages or the one or more vehicle control messages, a latency associated with at least one of the vehicle controller or at least one of the one or more processing systems;
determining, based at least in part on the latency being anomalous, an event associated with the autonomous vehicle; and
controlling, based at least in part on the event, the autonomous vehicle to operate in a safe state,
wherein the plurality of messages comprises:a first message including a first identification associated with a first instance of the first sensor data, first timestamp information associated with the first instance of the first sensor data, a second identification associated with a second instance of the second sensor data, second timestamp information associated with the second instance, a first system identification associated with a first processing system of the one or more processing systems, and first system timestamp information associated with the first processing system; and
a second message including the first identification, the first timestamp information, the second identification, the second timestamp information, a second system identification associated with a second processing system of the one or more processing systems, and second system timestamp information associated with the second processing system, and

wherein a vehicle control message of the vehicle control messages includes:first information including the first identification, the second identification, the first timestamp information, the second timestamp information, the first system identification, the first system timestamp information, vehicle controller identification information, and vehicle controller timestamp information associated with the vehicle controller, and
second information including the first identification, the second identification, the first timestamp information, the second timestamp information, the second system identification, the second system timestamp information, the vehicle controller identification information, and the vehicle controller timestamp information associated with the vehicle controller.



US Pat. No. 11,099,574

INTERNAL SAFETY SYSTEMS FOR ROBOTIC VEHICLES

Zoox, Inc., Foster City,...


1. A method comprising:receiving sensor data from at least one sensor associated with a vehicle;
determining a location of the vehicle within an environment, wherein the location identifies one or more of a position or an orientation of the vehicle in the environment;
identifying, based at least in part on the sensor data, an object within the environment;
determining, based at least in part on the sensor data, a classification associated with the object;
determining, based at least in part on the sensor data, a trajectory of the object;
determining, based at least in part on the classification and the trajectory of the object, an object type associated with the object;
determining a future location associated with the object based at least in part on the object type;
selecting a safety system of the vehicle based at least in part on the future location associated with the object; and
activating the safety system.

US Pat. No. 11,091,092

METHOD FOR ROBOTIC VEHICLE COMMUNICATION WITH AN EXTERNAL ENVIRONMENT VIA ACOUSTIC BEAM FORMING

Zoox, Inc., Foster City,...


1. A system comprising:an acoustic beam steering array comprising a plurality of speakers configured to output directional audio; and
one or more processors configured to perform operations, including:receiving a first signal indicative of one or more of an object type, an object position, an object trajectory, or a predicted intersection point of an object trajectory and a vehicle trajectory, the first signal associated with a first object;
determining, based at least in part on the first signal, a direction to emit a beam of acoustic energy via at least a portion of the plurality of speakers of the acoustic beam steering array;
receiving a second signal indicative of a second object, the second object being different than the first object;
determining, based at least in part on the first signal and the second signal, a first gain setting associated with a first portion of the beam of acoustic energy and a second gain setting associated with a second portion of the beam of acoustic energy, wherein the first gain setting is different than the second gain setting; and
causing the at least the portion of the plurality of speakers to emit the beam of acoustic energy indicative of an alert in the direction and at the first gain setting and the second gain setting,
wherein the first gain setting and the second gain setting are configured to cause a sound pressure of the beam of acoustic energy at the first object to be different from a sound pressure of the beam of acoustic energy at the second object.


US Pat. No. 11,091,113

VEHICLE SIDE AIRBAG

Zoox, Inc., Foster City,...


1. A side airbag system comprising:a chamber configured to be coupled to a side portion of a seat tub of a seat assembly of a vehicle, the chamber configured to mount between the seat tub and a seat pan of the seat assembly; and
an inflator coupled to the chamber, wherein the inflator is configured to receive a signal indicating a side impact and, based at least in part on the signal, cause the chamber to expand toward a centerline of the seat assembly and deform at least a portion of the seat assembly toward the centerline of the seat assembly,
wherein, when fully expanded, the chamber is occluded from a passenger compartment of the vehicle by the at least the portion of the seat assembly, and
wherein the chamber is configured to be coupled to the seat tub such that the seat pan is removable from the seat tub without decoupling or disarming the side airbag.

US Pat. No. 11,087,494

IMAGE-BASED DEPTH DATA AND LOCALIZATION

Zoox, Inc., Foster City,...


1. A system comprising:one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:accessing lidar-based map data of an environment, the lidar-based map data comprising a three-dimensional mesh associated with the environment;
capturing, by a sensor of an autonomous vehicle, image data associated with the environment;
inputting the image data to a machine-learned model;
receiving, from the machine-learned model, depth data associated with the image data, wherein the depth data comprising a point cloud associated with the environment; and
determining a location of the autonomous vehicle in the environment by comparing a first portion of the depth data to a second portion of the lidar-based map data.


US Pat. No. 11,085,224

VEHICLE DOOR ACTUATOR ASSEMBLY

Zoox, Inc., Foster City,...


1. A vehicle, comprising:a vehicle body defining a first interior space;
an actuator assembly connected to the vehicle body;
a vehicle door;
a housing defining a second interior space;
a base plate coupling the vehicle door to the vehicle body; and
a gear coupled to the base plate, the gear being driven by the actuator assembly to move the vehicle door between an open position providing access to the first interior space and a closed position substantially prohibiting access to the first interior space, wherein in the closed position the base plate is disposed substantially within the second interior space of the housing,
wherein the base plate is rotatable about an axis substantially perpendicular to the base plate to move the vehicle door along a travel path extending from the open position to the closed position while maintaining an outer surface of the vehicle door substantially parallel to a longitudinal axis of the vehicle along the travel path, wherein the gear comprising a first gear and is located proximal the vehicle body, the vehicle further comprising: a second gear coupled to the first gear, wherein rotation of the first gear causes rotation of the second gear to move the vehicle door relative to the base plate.

US Pat. No. 11,077,816

RESTRAINT SURFACE ADJUSTMENT SAFETY SYSTEM

Zoox, Inc., Foster City,...


1. A vehicle comprising:a bench seat to accommodate multiple occupants, the bench seat comprising a first seat space to accommodate a first occupant and a second seat space to accommodate a second occupant;
a first inflatable bladder associated with the first seat space to adjust at least one of a height or an angle of a first restraint surface of the first seat space;
a second inflatable bladder associated with the second seat space to adjust at least one of a height or an angle of a second restraint surface of the second seat space; and
an inflator disposed between the first inflatable bladder and the second inflatable bladder, the inflator being in fluid communication with the first inflatable bladder and the second inflatable bladder.