Painting Robot e-David

Project Overview

The e-David project merges painting with robotics and computer science. The primary objective of the project is to develop a painting machine that can replicate human artistic processes and to provide a framework for automatic painting. Since its inception in 2009, the project has made significant strides and has created paintings in a variety of styles and levels of abstraction, including realistic portraits and abstract compositions. Some of the artwork created by e-David has even been showcased in exhibitions.

The robot comprises an industrial robot arm fitted with a paintbrush, providing six degrees of freedom for movement, which enables it to apply paint to the canvas from different angles. The robot is equipped with a camera that provides visual feedback during the painting process, and a control computer to enable dynamic behaviour. The machine can use different pallets and brushes to create different painting styles and mediums, such as acrylics, ink, gouache, and up to 81 different colours. The robot can also use a variety of paintbrushes, ranging from simple brushes to large Chinese calligraphy brushes. These can also be automatically cleaned between uses.

The larger IRB 1660ID robot is a fixed installation in our robotics lab, while the smaller IRB 1200 by ABB can be moved for exhibitions. The smaller robot comes with a paper feed, which is helpful in creating larger images by pulling paper over the canvas using a roller mechanism powered by a stepper motor. Additionally, two plotters are available for student projects, one of which is desktop-sized and easily portable, in case the robots are not available.

The overall painting process works as follows:

First, an image is decomposed into shapes using a custom image abstraction procedure. This relies on an AI model to provide us with semantic information, allowing the robot to cleanly separate objects. Given these shapes, our software generates an overall paint plan to realize all shapes and classifies them as regions, single strokes or details. For each of these a style is chosen in which it should be realized, based on image smeantics and geometry. Then, each detected feature is painted onto the canvas using visual feedback, i.e. a camera supervises the canvas and detects errors which are iteratively corrected.

Student Projects

Several student projects have been completed or are currently in development in combination with the robot:

• Automatic High-Speed Portraiture (Emily Bihler, Bachelor + Master)
• Image Abstraction (Michael Stroh, Master)
• Writing of Unicode Characters (Silvan Zingrebe, Bachelor)
• Robot Path Debugging and Brushtroke Analysis (Fabio Cicetti, Bachelor)
• Gradient Painting (Jenifer Durai, Master)
• Automatic Stippling using AI Models (Raphael Aronowski, Bachelor)

Challenges & Goals

One of the main challenges of the e-David project is the unpredictability of the paintbrush, which can cause occasional variations due to the build-up of paint on the brush over time and the texture of the paper. Although the repeatability of a brushstroke can be precise up to +- 2mm, the nature of the brush makes it difficult to achieve human-like adaptation of the brush directly as the stroke is being painted. 

Additionally, there is no pressure detection, so the pressure limit for each brush type needs to be measured and stored. 

Safety is also an important aspect of the project, and adding new features may result in adjusted safety regulations being adopted. It is not possible to stand directly next to the robot during automatic work.

The e-David project has several goals for the future. 

Hardware flexibility is one of the objectives, and the project currently runs on two robots and a plotter. 

The team also aims to create an open interface to make the project more accessible to other researchers as well as providing usability and user friendliness in order to facilitate collaboration with artists. Such as developing a modular painting process that can combine different styles of painting processes.

The project also plans to focus on machine learning, and the team is exploring different approaches to include it. Currently the main Idea is to implement it in the paint plan creations and stroke modulations. Such as organising the subject matters of the reference image into paintable constructs and the effective creation of brushstrokes learned from previous interactions of painting.