Alphagalileo > Item Display

X-ray street vision

06/09/2021 Osaka University

Researchers at Osaka University create a custom dataset of building facades to train a machine learning algorithm to digitally remove unwanted objects, which may lead to advancements in automatic image reconstruction technology

Osaka, Japan – Scientists from the Division of Sustainable Energy and Environmental Engineering at Osaka University used generative adversarial networks trained on a custom dataset to virtually remove obstructions from building façade images. This work may assist in civic planning as well as computer vision applications.

The ability to digitally “erase” unwanted occluding objects from a cityscape is highly useful but requires a great deal of computing power. Previous methods used standard image datasets to train machine learning algorithms. Now, a team of researchers at Osaka University have built a custom dataset as part of a general framework for the automatic removal of unwanted objects — such as pedestrians, riders, vegetation, or cars — from an image of a building’s façade. The removed region was replaced using digital inpainting to efficiently restore a complete view.

The researchers used data from the Kansai region of Japan in an open-source street view service, as opposed to the conventional building image sets often used in machine learning for urban landscapes. Then they constructed a dataset to train an adversarial generative network (GAN) for inpainting the occluded regions with high accuracy. “For the task of façade inpainting in street-level scenes, we adopted an end-to-end deep learning-based image inpainting model by training with our customized datasets,” first author Jiaxin Zhang explains.

The team used semantic segmentation to detect several types of obstructing objects, including pedestrians, vegetation, and cars, as well as using GANs for filling the detected regions with background textures and patching information from street-level imagery. They also proposed a workflow to automatically filter unblocked building façades from street view images and customized the dataset to contain both original and masked images to train additional machine learning algorithms.

This visualization technology offers a communication tool for experts and non-experts, which can help develop a consensus on future urban environmental designs. “Our system was shown to be more efficient compared with previously employed methods when dealing with urban landscape projects for which background information was not available in advance,” senior author Tomohiro Fukuda explains. In the future, this approach may be used to help design augmented reality systems that can automatically remove existing buildings and instead show proposed renovations.


The article, “Automatic object removal with obstructed façades completion using semantic segmentation and generative adversarial inpainting” was published in IEEE Access at DOI:
Title: “Automatic object removal with obstructed façades completion using semantic segmentation and generative adversarial inpainting”
Journal: IEEE Access
Authors: Jiaxin ZHANG, Tomohiro FUKUDA, and Nobuyoshi YABUKI
DOI: 10.1109/ACCESS.2021.3106124
Funded by: Japan Society for the Promotion of Science
Attached files
  • Fig.1 Results of automatic object removal and facade completion. (a) People, (b) rider, (c) vegetation, (d) car. (credit: © 2021 Jiaxin ZHANG et al., IEEE Access)
  • Fig.2 Wokflow for automatic object removal and obstructed facade completion using semantic segmentation and generative adversarial inpainting. (credit: © 2021 Jiaxin ZHANG et al., IEEE Access)
  • Fig.3 Collection method for a perpendicular street facade. (a) Road networks, (b) sampled points from the centerline of the road, (c) deflection angle θ, and (d) orthographic facade obtained from a street view service. (credit: © 2021 Jiaxin ZHANG et al., IEEE Access)
06/09/2021 Osaka University
Regions: Asia, Japan
Keywords: Applied science, Computing, Engineering


For well over a decade, in my capacity as a researcher, broadcaster, and producer, I have relied heavily on Alphagalileo.
All of my work trips have been planned around stories that I've found on this site.
The under embargo section allows us to plan ahead and the news releases enable us to find key experts.
Going through the tailored daily updates is the best way to start the day. It's such a critical service for me and many of my colleagues.
Koula Bouloukos, Senior manager, Editorial & Production Underknown
We have used AlphaGalileo since its foundation but frankly we need it more than ever now to ensure our research news is heard across Europe, Asia and North America. As one of the UK’s leading research universities we want to continue to work with other outstanding researchers in Europe. AlphaGalileo helps us to continue to bring our research story to them and the rest of the world.
Peter Dunn, Director of Press and Media Relations at the University of Warwick
AlphaGalileo has helped us more than double our reach at SciDev.Net. The service has enabled our journalists around the world to reach the mainstream media with articles about the impact of science on people in low- and middle-income countries, leading to big increases in the number of SciDev.Net articles that have been republished.
Ben Deighton, SciDevNet

We Work Closely With...

  • BBC
  • The Times
  • National Geographic
  • The University of Edinburgh
  • University of Cambridge
Copyright 2021 by DNN Corp Terms Of Use Privacy Statement