TRENDS
12
plate and slice the pizza; then deposit a slice on
the plate. This series of actions requires grasps of
hard, soft, and even floppy objects, hot and cold
objects, as well as liquids and solids. While humans
instinctively understand how to grasp any object—
even one we’ve never seen before—robots have
to be taught this skill: The robot has to perceive
the object with its sensors, model it appropriately,
determine a strategy for picking it up, then execute
the desired action. The extensive training required
to teach commercial robots these skills is expensive.
That’s where AUTOLAB’s work comes in.
LEARNING THROUGH FAILURE
Goldberg, Mahler, and their post-graduate students
began working on AUTOLAB’s Dexterity Network
(Dex-Net) in 2015. The groundbreaking venture
develops and refines robot “picking” strategies
and, just as importantly, has improved the machine
learning behind the picking calculations.
“Dex-Net can be used to train a robotic system
for handling a variety of items without advance
knowledge,” says Mahler, citing CAD models, mass,
or images as examples. “One of the advantages is
that it can be rapidly adapted to different hardware
systems consisting of various arms, grippers, and
3D depth cameras, enabling faster customization
of robotic learning systems.”
The first iteration of Dex-Net entailed a system for
grasping one object at a time with parallel jaws—think
two fingers or pliers. Their current work—Dex-Net
4.0—trains robots to grasp a wider variety of objects
piled in heaps that make picking more challenging.
Dex-Net 4.0 includes both the parallel-jaw gripper and
a pneumatic suction arm—each with its own neural
network. The robot’s central programming provides
size and shape information via sensors, but the two
arms’ separate neural networks decide whether an
object should be handled by grip or suction.
While AUTOLAB researchers applaud their
advances, what really interests them is the failures:
the objects the robot couldn’t pick up or hold on to.
“Part of the AUTOLAB philosophy is to probe
for failure modes that provide deeper insight into a
method,” Mahler says. “That’s where the adversarial
objects came from. The results behind Dex-Net
were only possible with countless hours of meticulous
experimentation and healthy skepticism.”
AUTOLAB has designed and created thousands
of adversarial objects—some as virtual simulations
and many others 3D-printed. Some objects look
like familiar shapes, but with a peculiar twist—like
a cube where part of one surface has been shaved,
creating a new plane that easily slips out of the
grippers of a cube-picking robot. Others seem
surfaced from a nightmare: melted, twisted fivelegged
objects. The physical objects are small—
around 10cm—as they’re meant to thwart a robot
with 5cm grippers, Mahler says.
AUTOLAB’s robot uses its suction arm to pick up scissors.
QUICKER PICKING
Perhaps AUTOLAB’s biggest breakthrough is that
Goldberg, Mahler, and their adversarial objects