paper-tex/sections/introduction.tex

%auto-ignore

\section{Introduction}

% Distractor removal is a common step in photo editing. Conventionally, to remove any unwanted objects from an image, the user first draws a mask over the object, then uses a hole-filling algorithm to remove them and fill in the background. With recent advance in semantic segmentation, automatic masking of some common objects is possible. Among the types of distractors, wires are one of the most common yet unique objects to remove in photographs. Their ubiquity and unique attributes calls for extra attention and importance to perform automatic wire segmentation.

Oftentimes wire-like objects such as powerlines and cables can cross the full width of an image and ruin an otherwise beautiful composition. Removing these ``distractors'' is thus an essential step in photo retouching to improve the visual quality of a photograph. Conventionally, removing a wire-like object requires two steps: 1) segmenting out the wire-like object, and 2) removing the selected wire and inpainting with plausible contents. Both steps, if done manually, are extremely tedious and error-prone, especially for high-resolution photographs that may take photographers up to hours to reach a high-quality retouching result.
%The second step has benefited from existing image inpainting techniques (e.g. Content-aware Fill~\cite{barnes2009patchmatch} in Photoshop) and can be accomplished in an automatic way. The first step -- wire mask creation -- is much more manually intensive and error-prone, and automatic methods have yet to be fully explored.

% In this paper, we address the problem of wire masking as a way to automate the step of identifying and segmenting out those objects. 
In this paper, we explore a fully-automated wire segmentation and inpainting solution for wire-like object segmentation and removal with tailored model architecture and data processing.
For simplicity, we use \textit{wire} to refer to all wire-like objects, including powerlines, cables, supporting/connecting wires, and objects with wire-like shapes.

Wire semantic segmentation has a seemingly similar problem setup with generic semantic segmentation tasks; they both take in a high-resolution image and generate dense predictions at a pixel level.
However, wire semantic segmentation bears a number of unique challenges. First, wires are commonly long and thin, oftentimes spanning the entire image yet having a diameter of only a handful of pixels. A few examples are shown in Figure~\ref{fig:motivation}. This prevents us from getting a precise mask based on regions of interest. Second, the input images can have arbitrarily high resolution up to 10k$\times$10k pixels for photographic retouching applications. Downsampling such high-resolution images can easily cause the thin wire structures to disappear. This poses a trade-off between preserving image size for inference quality and run-time efficiency.
Third, while wires have regular parabolic shapes, they are often partially occluded and can reappear at arbitrary image location, thus not continuous.
%This limits the use of parametric modeling 
(e.g.~\cite{lanedet,swiftlane}).
%Segmentation predictions on downsampled images capture fewer details, whereas inference on high-resolution images requires quadratically more GPU memory that may be too big to fit into a modern GPU. 
% We also note that recent semantic segmentation methods that specifically target high-resolution images also have their limitations when applied to wire images.
% We summarize the challenges of wires in Figure~\ref{fig:motivation}.

To account for these challenges, we propose a system for automatic wire semantic segmentation and removal. For segmentation, we design a two-stage coarse-to-fine model that leverages both pixel-level details in local patches and global semantics from the full image content, and runs efficiently at inference time. For inpainting, we adopt an efficient network architecture~\cite{suvorov2022resolution}, which enables us to use a tile-based approach to handle arbitrary high resolution. We design a training strategy to enforce color consistency between the inpainted region and the original image.
We also present the first benchmark dataset, \benchmark, for wire semantic segmentation tasks, where we collect and annotate high-resolution images with diverse scene contents and wire appearances.
We provide analyses and baseline comparisons to justify our design choices, which include data collection, augmentation, and our two-stage model design. Together, these design choices help us overcome the unique challenges of accurately segmenting wires.
% We report our model performance and efficiency on this benchmark dataset, and provide a set of comparisons with popular semantic segmentation methods on this task.
Our contributions are as follows: 
% zwei{reordered the items}

\begin{itemize}[noitemsep]
  %\item We introduce a novel task: wire semantic segmentation for high-resolution photographic images, an under-explored task that is extremely important for applications like photo retouching.
  %\item We introduce a wire semantic segmentation benchmark dataset that consists of photographic images at high resolution, with large scene diversity, various types of cameras, a rich collection of wire shapes and high-definite manual annotations.
  \item \textbf{Wire segmentation model:} We propose a two-stage model for wire semantic segmentation that leverages global context and local information to predict accurate wire masks at high resolution. We design an inference pipeline that can efficiently handle ultra-high resolution images.
  \item \textbf{Wire inpainting strategy:} We design a tile-based inpainting strategy and tailor the inpainting method for our wire removal task given our segmentation results.
  \item \textbf{\benchmark, a benchmark dataset:} We collect a wire segmentation benchmark dataset that consists of high resolution images, with diversity in wire shapes and scene contents. We also release the manual annotations that have been carefully curated to serve as ground truths. Besides, we also propose a benchmark dataset to evaluate inpainting quality. 
  %\yq{Explain the most important difference between ours and previous existing datasets like diversity of camera viewpoints or wire width etc. (mt:changed)}
 %\yq{What is the advantage of the newly proposed model? Will it be helpful for processing ultra-high resolution images? Is the global information used efficiently? (mt:changed)}
  %\item We describe a set of domain-specific data augmentation and inference pipelines specifically for wire-like objects, which help overcome unique challenges in this task and improve model performance.\yq{Do we have the ablation study to show the data augmentation improves the performance? If we want to claim it as contribution, it should have some benefits and novelty in the method itself. (mt: removed)}
\end{itemize}

%Distractor removal is a common component in photographic enhancement. Among the various distractors, wire is one of the most common and general types. Automatically detecting and segmenting wires in an image will greatly reduce the manual masking work. However, 
% Conventionally, the user manually draws a mask over unwanted objects, then uses hole-filling algorithms to remove them. Recent advances in deep learning introduced object segmentation methods that automatically draw masks over objects of interest, thereby automating the masking process. Nevertheless, 
% there are several unique challenges in training wire segmentation models: (a) the unique shape properties of wire (thin) makes it need special treatment such as resizing and sampling. This renders the use of common object segmentation methods in this task difficult, and require task-specific designs in model and training schemes. (b) it is hard to collect data (this is the reason why we have the synthetic data. (c) there is a lack of standard dataset to benchmark the progress.  

 %\textcolor{blue}{flow: distractor removal (common operation) -> manual masking -> automating masking (segmentation), among all distractors, wire unique and common and important and general -> automatkcill segment wire is important}
 
 
  %\textcolor{blue}{wire seg bears similar to semantic segmentation in that (a, sinput, b..), but due to the attributes (thin, large span,,,) of wire, normal segmentation is suboptimal. to ward this end, we developped xxx that addresses these issues  our models is a two stage corease file, maintains efficenty}.
 
 % \textcolor{blue}{Moreover, we provide a systemactic frameworks from data data processing ,data ayucmetion to post-process}.
 
 
 % \textcolor{blue}{realtive new field, wihtout too much data, previous benches marks (low...), we contrbute to this reserach field wtih a high quality benchmark, we also profiled a couple of metrics and discuss the performance }
  
  %\textcolor{blue}{to sum up, our xxx 1, 2. 3}
  
  
  
 
 % flow: 
 
%\textcolor{red}{REARANGE SECTIONS TO ONE SMOOTH STORY}

%\subsection*{Conventional semantic segmentation}
%In the conventional setting of semantic segmentation, the model learns to segment entities such as person, table and car. A common trait of these objects of interest is that they are somewhat regularly shaped, which means the object's width and height are similar. This property is very advantageous in terms of model training, since the object features will remain equally prominent when the image is resized without distortion. As a result, it is a common approach to downsample a large image to fit within a GPU for training and inference without losing object details.

%\subsection*{Segmenting wires in outdoor images}
%Outdoor wires and cables exhibit significantly different shape and lighting properties than common objects. First, many wires in outdoor images are extremely long and thin. This means that wires can often span across the entire image, while being only a few pixels thick. This poses several problems to performing inference on images, since too much downsampling would cause the wire to disappear, while too little downsampling would impose significant computation overhead. Motivated by these challenges, we propose a pipeline for wire segmentation in high-resolution images for image enhancement and provide a set of baseline in this task. Our contributions are as follows:

%In this paper we systematically tackle the problem of wire segmentation and address issues including data collection, image pre-processing and data augmentation during training.  

% \section{Introduction}

% Distractor removal is a common component in photographic enhancement. Conventionally, the user manually draws a mask over unwanted objects, then uses hole-filling algorithms to remove them. Recent advances in deep learning introduced object segmentation methods that automatically draw masks over objects of interest, thereby automating the masking process. Nevertheless, there are several unique challenges in wire segmentation that renders the use of common object segmentation methods in this task difficult, and require task-specific designs in model and training schemes.

% \subsection*{Conventional semantic segmentation}
% In the conventional setting of semantic segmentation, the model learns to segment entities such as person, table and car. A common trait of these objects of interest is that they are somewhat regularly shaped, which means the object's width and height are similar. This property is very advantageous in terms of model training, since the object features will remain equally prominent when the image is resized without distortion. As a result, it is a common approach to downsample a large image to fit within a GPU for training and inference without losing object details.

% \subsection*{Segmenting wires in outdoor images}
% Outdoor wires and cables exhibit significantly different shape and lighting properties than common objects. First, many wires in outdoor images are extremely long andthin. This means that wires can often span across the entire image, while being only a few pixels thick. This poses several problems to performing inference on images, since too much downsampling would cause the wire to disappear, while too little downsampling would impose significant computation overhead. Motivated by these challenges, we propose a pipeline for wire segmentation in high-resolution images for image enhancement and provide a set of baseline in this task. Our contributions are as follows:

% \begin{itemize}
%   \item We introduce a wire semantic segmentation dataset consisting of high-resolution images with wires.
%   \item We propose a two-stage model for wire segmentation that captures high-resolution image features without significant computation overhead.
%   \item We describe a set of pre-processsing and post-processing pipelines specific for wires, which helps overcome unique challenges in this task.
% \end{itemize}