CGI 2020

Deep Color Transfer using Histogram Analogy

Junyong Lee1, Hyeongseok Son1, Gunhee Lee2, Jonghyeop Lee1, Sunghyun Cho1, Seungyong Lee1
1POSTECH 2NC Soft
Paper Supp Video Code
overall framework

We propose a novel feed-forward network for image color transfer
that leverages deep encoded histogram features.

Abstract

We propose a novel approach to transferring the color of a reference image to a given source image. Although there can be diverse pairs of source and reference images in terms of content and composition similarity, previous methods are not capable of covering the whole diversity. To resolve this limitation, we propose a deep neural network that leverages color histogram analogy for color transfer. A histogram contains essential color information of an image, and our network utilizes the analogy between the source and reference histograms to modulate the color of the source image with abstract color features of the reference image. In our approach, histogram analogy is exploited basically among the whole images, but it can also be applied to semantically corresponding regions in the case that the source and reference images have similar contents with different compositions. Experimental results show that our approach effectively transfers the reference colors to the source images in a variety of settings. We also demonstrate a few applications of our approach, such as palette-based recolorization, color enhancement, and color editing.

Relevance between Source and Reference Images

We first divide various correlations between source and reference images into three cases. The first case is the strong relevance, where two images have high similarity both in the contents and positions of semantic objects. Second, the weak relevance refers to high similarity in the contents but with less correlations in the object spatial configurations. The last case of irrelevance includes image pairs with dissimilar contents and special settings with graphic design images and color palettes as the reference.

relevance

Histogram Analogy

In our approach, histogram analogy is exploited either uniformly or adaptively on the source image, depending on the relevance of an input pair. For strongly relevant and irrelevant cases, the same histogram information is used for all parts of the source image, and this is the default setting. When semantic object information is important, as in the case of weak relevance, we can use semantic image segmentation [25], and the histogram analogy is extracted and applied for corresponding semantic regions between source and reference images. With the default setting and the variation with semantic segmentation, our approach can support both global and local color transfers, covering the whole diversity of input image pairs.

HA
SR

Segment-wise Semantic Replacement

When the source and reference images are weakly relevant with high semantic similarity, but the low correlation in object configuration, globally applying histogram analogy would not always generate visually pleasing results. This is predictable as no clue of semantic relationship could be obtained by just looking at the globally computed source and reference histograms. In that case, we explicitly provide additional semantic information to the network with semantic replacement.

Results

Qualitative Comparison

Ablation

User Study (Amazon Mechanical Turk)

Ablation

Applications

BibTeX

@Article{Lee2020CTHA,
    author  = {Junyong Lee and Hyeongseok Son and Gunhee Lee and Jonghyeop Lee and Sunghyun Cho and Seungyong Lee},
    title   = {Deep Color Transfer using Histogram Analogy},
    journal = {The Visual Computer},
    volume  = {36},
    number  = {10},
    pages   = {2129--2143},
    year    = {2020},
}