The hope is that, in the future, robots will be able to learn autonomously to execute a multitude of different movements, which is something we humans can do without giving it a second thought: But do we actually need the complicated human musculoskeletal system or are muscles more of an evolutionary happenstance? Researchers led by Professors Marc Toussaint and Syn Schmitt are investigating whether an intelligent robotic arm can learn movements faster and easier with the aid of artificial muscles.
Reach for a glass, write a letter, throw and catch the ball, walk, hop or balance: man is undoubtedly an all-rounder. Robots that have mastered individual movements, such as baking pancakes, unscrewing a bottle or jumping, impressively well already exist. However, there is currently not a single robot in existence that could balance on a slackline, for example, or that would have the necessary sensitivity to swipe a smartphone screen with its finger. “That sounds trivial, but such movements are not so easy to control,” says Marc Toussaint, head of the Machine Learning and Robotics group at the Institute of Parallel and Distributed Systems (IPVS) at the University of Stuttgart. When swiping, for example, the finger must detect the surface and keep in constant contact with it. “The robot finger would quickly press too hard and then more or less stumble,” explains the expert for artificial intelligence, imitating the forwards-tapping movement of the robot finger with his finger.
In the “Deep Control” project, which is coordinated by Toussaint, physicists, computer scientists and control engineers are investigating whether muscles could help to better control the movements of robots to enable them to perform complicated movements one day. The test objects are computer simulations and a muscle-bound robotic arm that is designed to learn autonomously to point in any given direction. The Baden-Württemberg Foundation has been funding the project for a total of three years since the spring of 2017 as part of its Neurorobotics research program. Both physicists are also active in Baden-Württemberg’s “Cyber Valley” initiative, the objective of which is to promote research into artificial intelligence. Until now, the limbs of humanoid robots have usually been rotated and flexed directly by electric motors, based on industrial robot models. Although today's robots can already point in any direction with extreme precision, the effort involved is high. “To move a robotic arm from one point to the next, the controller must send new signals to the motor every millisecond,” says Toussaint. Parallel sensors continuously monitor the arm’s current location. The human brain has it easier in this respect: All it takes, Toussaint explains, is for a nerve impulse to be sent to the arm muscles, whereupon the muscles contract and the arm automatically executes the movement.
Unpredictable muscle play
If robots are equipped with artificial muscles in the hope that they will be able to carry out more difficult movements in the future, they will first have to learn the relevant movements, just as humans do. Human movements involve a complex interplay between different brain regions, different muscle groups that relax or contract in a precisely coordinated manner, and tendons, which transfer muscle power to the bones. “The control algorithms of muscle-driven robots have no other choice than to learn the requirement movement,” says Toussaint. “If I use motors to control the joints directly, I can then use the joint angles to calculate where the finger is and then use an algorithm to bring the finger into a desired position; that's simple mathematics,” says Toussaint. But, he continues, if one wants to introduce muscles as a further level of regulation, there will be no mathematics. It is no coincidence that the name of the project is “Deep Control”, which means nothing other than deep control of the movement across several hierarchical levels.
Two master’s students of computer science from Toussaint's research group have recentlly programmed the “brain” for a robotic arm, an intelligent control algorithm based on machine learning. Syn Schmitt's recently renamed “Computational Biophysics and Biorobotics” research group has, so to speak, contributed the artificial muscles, tendons and bones. On the one hand, the robotic arm exists purely virtually as a simulation model on the computer. In addition, Schmitt, co-director of the new Institute for Modelling and Simulation of Biomechanical Systems, has had a real life-size torso with a single right arm built.
The metal robotic arm can be moved via a shoulder and elbow joint and terminates in a rigid hand with an outstretched index finger. The “arm muscles” are actually cylinders made of a pressure-tight rubber hosing with an internal yarn lining, and are operated pneumatically: as soon as a compressor pumps compressed air into the hose, it expands and becomes thicker and shorter like a natural muscle, i.e., it contracts. A total of five of these artificial muscles enable the robotic arm to move up and down as well as backwards and forwards on a single plane.
The muscles interact in a similar way to their human models: a “biceps” muscle on the upper side of the upper arm attaches to the elbow joint and bends the arm, its opponent, the “triceps” muscle on the back of the upper arm, stretches the arm again. Two antagonistic muscles in the torso move the shoulder joint based on the same principle. Another muscle in the torso can both move the arm forward and bend it at the shoulder joint. In addition, Schmitt's employees have installed finger-thick springs as tendons that protrude from the top of the shoulder joint, as well as various sensors that measure the force exerted by and length of the artificial muscles as well as the angle of the joint and transmit them to the computer.
Robot learns from experience like a baby
But how does the robot’s artificial intelligence, or more precisely the control algorithm behind it, learn to let the arm point in a particular direction? To put it simply, it learns like a human baby that initially fidgets around in an uncoordinated manner and eventually moves on to more coordinated movements based on its experiences. “Initially, the algorithm has no idea what signals it needs to send to the muscle to move the arm to a certain point,” Toussaint explains. So it sends any random signal, and the arm might move down. It then saves this experience as a data point. From this first data point it generates a rough model of how to move the arm up or down. Back in the starting position, the next thing it might do could be to send a signal in the opposite direction and, sure enough, the arm does actually move in the opposite direction. This is the second data point. “That won't yet get the arm to the right destination, but it may well get a lot closer,” says Toussaint.
And so the artificial neural network is trained incrementally and the model it constructs through the learning experience becomes more and more accurate: up to 700 times, in the simulation even more than 1,000 times, always a new pointing movement. The virtual “map” is gradually filled up with data points and the necessary muscle signal is assigned to every possible target that the arm has aimed at. “If the arm wants to go down in the future, it will know how to do so because it has already experienced 'down',” Toussaint explains. This learning method is based on so-called deep learning, which has been used for years in image recognition and in the speech recognition module used by Siri, the digital assistant on the iPhone. This involves arranging a network of nodes corresponding to the neurons in the brain into levels that use increasingly complex features to recognize something such as a face. For example, the first layer will only musidentify the brightness values of pixels. The second layer would register that some pixels are associated with edges. The next one would distinguish between horizontal and vertical lines, until at some point a level is reached on which the system learns which shapes are suitable for face recognition. In Reinforcement learning, connections between the nodes are either reinforced or weakened according to the carrot and stick principle, depending on the experiences made after each training round.
“We asked ourselves what the analogue of this idea of signal processing is when it comes to controlling a system,” says Toussaint, “What does it mean for a control problem to introduce more and more hierarchical levels? So far, he continues, there are hardly any concepts available for controlling robot movements, only for computer games at most, in which a game character learns to walk through a labyrinth or carry out certain actions. As a major innovation, Toussaint's researchers have also equipped the robot's “brain” with an exploration strategy so that the robotic arm systematically explores its entire range of motion during training. “If the algorithm next selects a target that is very distant from all previous targets, it will learn just as little from the data as if it were to move towards the same destination umpteen times,” explains Toussaint. The program code that the researchers have installed, on the other hand, stipulates that the robotic arm always selects targets between the two extremes - distant from previous experience, but not too far away. “This guarantees that the algorithm learns the most from the experience it has gained,” says Toussaint.
“From an engineer's point of view, absolute nonsense.”
Now, if Schmitt presses the button of a remote control at his institute, the robotic arm moves purposefully in the desired position, albeit swaying slightly. “The deviation for a target point is about eight millimeters, and in the simulation it is even less than two millimeters,” he reports and hopes that a corrective camera could increase precision even further. But what drives the biophysicist most is the question of why biology has produced such complicated things as muscles. Was the human musculoskeletal system driven by evolution, or simply by the fact that there are no other building materials available? “From an engineering perspective,” Schmitt goes on, “it's absolute nonsense to install two motors around a joint , i.e. antagonistic artificial muscles, rather than a single electric motor that can rotate in both directions”.
The team have now answered this question themselves by first having the robotic arm learn the pointing movement once using muscle power and once using direct joint control via an electric motor and comparing the results. In fact, muscles and tendons make it easier to regulate the movement. “Easier means easier to understand and faster to learn, which means that the algorithm needs significantly fewer data points for training. It also means less computing effort, i.e. smaller processors, resulting in increased energy efficiency,” Schmitt concludes.
In a computer simulation of a jumping robot , a former doctoral research student of Schmitt’s had previously found that much less information needs to be exchanged between the sensors, the control algorithm and the drive system, and computed, when the robot is driven by muscles. He devised the metric used to measure the control effort required for movements specially for this purpose. In the current project, however, the researchers also tested what happens when the arm's movement is disturbed by someone pressing against it: Will the robotic arm still be able to steer towards its target with precision after that? “We can clearly see that what was learned at the muscle level is much more robust than what was learned directly at the joint level,” Toussaint reports.
The next objective is for the robot to learn to master a really difficult task, i.e., swiping a finger across a surface. However, Toussaint and Schmitt point out that there is still a long way to go before their research results will be ready for use in a marketable robot. “Many challenges still remain” says Schmitt: “there’s the limited installation space in which many mechanical parts have to be accommodated; the pneumatic muscles that need a giant compressor; the robustness of the drive system. “Basically, we would now have to test which muscle models work particularly well,” adds Toussaint. It is possible that after optimization “the muscles” would then no longer resemble real muscles at all.
Side by side with the robot of the future
In the future, the muscle-bound, autonomously learning robot could be used when natural, varied movements are required, or when the robot has to adapt the force with which it executes the movement to unforeseen events. Collaborative workplaces in which robots and humans work side by side could be one such area, for example. If a human and robot were to accidentally collide, the robot would give way and take evasive action, which is something an industrial robot cannot do. “I am convinced that a muscle-driven robot can carry out such evasive actions much more easily,” says Schmitt. More natural exoskeletons are also conceivable, giving workers “super powers”, for example for overhead installations or lifting heavy loads. In rehabilitation, they could help stroke patients or paraplegics learn to walk again. “The exoskeleton would learn, together with the carrier, to execute the movement in an adequate manner,” says Schmitt. Humans are still far superior to robots. The next step towards the intelligent robots of tomorrow that Toussaint wants to climb in the coming years entails “physical thinking”. For example, a robot that is supposed to grab a wooden block but can't reach it might try to throw a ball that was within its reach against the wall so that it bounces at the right angle needed to move the wooden block within reach of the robot. To loosely quote Neil Armstrong, this task is one small step for a man, one giant leap for robotics.