Deriving Semantic Contour Lines for Robotic Painting in Natural Images and Portraits
Project Overview:
This project explores the derivation of semantic contour lines from natural images and portraits, enhancing their representation for digital, non-photorealistic rendering, and robotic painting applications. Contour lines are a powerful tool in art, used to define the shapes and boundaries of objects. By leveraging semantic information, students will develop techniques to extract meaningful contours that capture an image's structural and abstract elements. These contour lines will be used in the E-David robotic painting system to produce physically stylized artistic renderings that emphasize key features of a scene or portrait. Since artistic stylizations can be highly subjective and contour line detection is a complex problem, involving the user in an interactive process is often essential to edit, add, and delete parts of the automatically derived contours to enhance the results further.
Objectives:
The primary objective is to create a pipeline that integrates image segmentation, semantic analysis, and edge detection in an interactive system to generate abstracted contour lines. This involves a unique and challenging combination of computer vision techniques, such as semantic segmentation and contour extraction, with artistic principles of stylization and abstraction. The contours derived will highlight significant boundaries, while selectively ignoring redundant or less important details, enabling the robotic painting system to produce visually appealing outputs with minimal intervention during the painting process. The result will be a flexible system that can adapt to various styles and levels of abstraction and might be applied in conjunction with an existing region-based image abstraction framework to create composite paintings.
Methodology:
The project will begin by investigating existing methods for interactive edge detection and semantic segmentation, focusing on merging these techniques to extract semantically meaningful and artistically expressive contours. Panoptic segmentation ensures that the system considers object instances and their broader semantic categories in contour detection. Additionally, saliency maps guide the system in identifying the most essential regions for retaining detail. Students are encouraged to create an interactive web-based contour editing framework to change, add, or remove parts of the automatically detected contours to enable users to correct mistakes or add personal highlights to the contour results. We can provide a framework for a web-based interface written in Svelte.
Robotic Implementation:
The project's second phase will involve sending the derived contours to a robotic painting system and painting on actual canvases. The robotic arm will be programmed to follow the contours, using varying brush strokes, pressure, and colors according to the semantic importance of each line. The contours will serve as both a structural guide and a stylistic inspiration, allowing the robot to produce paintings that range from detailed, realistic representations to highly abstract forms. The project will not necessarily involve close hardware integration, as a framework for sending SVG-based shapes to the robot is already working. However, for willing students, the project can also move to more hardware-based work focusing on accurately following complex contour or stroke paths derived from images with the robotic system.
Expected Outcomes:
The project aims to produce a flexible framework for interactively generating semantic contour lines applied to various artistic styles and media. The robotic painting system will demonstrate the ability to autonomously create both naturalistic and abstract representations of natural images and portraits, using contour lines as the primary guide. The outcome will not only advance the field of robotic painting but also offer new possibilities for human-robot collaboration in artistic endeavors, where robots assist in producing complex, stylized visual representations.
Requirements:
General interest in image processing, robotics, working with semantic information
Basic knowledge of Python programming and image processing
Willingness to take an artistic perspective
If you are interested, please contact Michael Stroh.