Style Transferring Of Image Using (CycleGAN)

Introduction

Cycle-Consistent Adversarial Networks CycleGANs were proposed by Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. CycleGANs are a novel approach for translating an image from a source domain A to a target domain B. One of the cool feature of CycleGANs is that it doesn’t require paired training data to produce stunning style transfer results.

In many style transfer applications, paired data is a required for the training.

CycleGAN is doesn’t require paired data input to train a models.

A CycleGAN tries to learn a Generator network, which, learns two mappings. CycleGANs train two Generators and two Discriminators networks. which differs from most of the GANs with a single Generator and Discriminator network.

DDI Editor's Pick: 5 Machine Learning Books That Turn You from Novice to Expert - Data Driven…

CycleGAN has two Generator networks

• Generator A: Learns a mapping G:X ->Y, where X is an image from the source domain A and Y is an image from the target domain B. It takes an image from the source domain A, and converts it into an image that is similar to an image from the target domain B. Basically, the aim of the network is to learn a mapping so that G(X) is similar to Y.
• Generator B: Learns a mapping F:Y->X, and then takes an image Y from the target domain B, and converts it into an image X that is similar to an image from the source domain A. Similarly, the aim of the network is to learn another mapping, so that F(G(X) is similar to X.

CycleGAN has two Discriminator networks

• Discriminator A: The Discriminator A is to Discriminate between the images generated by the Generator network A, which are represented as G(X), and the real images from the source domain A, which are represented as X.
• Discriminator B: The Discriminator B is to Discriminate between the images generated by the Generator network B, which are represented as F(Y), and the real images from the source domain B, which are represented as Y.

CycleGAN’s Architecture Diagram

The goal of CycleGANs is to learn how to map images from one domain A to another domain B. CycleGAN’s architecture diagram shows the use of two mapping functions G and F, and two Discriminators DX and DY.

• The discriminator DX is used to verify the images from domain A and the translated images F(y).
• The discriminator DY is used to verify the images from domain B and the translated images G(x).
• The CycleGANs use a cycle consistency loss for image translation. The cycle consistency loss is used to determine whether an image X from domain A can be recovered from the translated image.

X to Y then reconstructed X CycleGAN’s Architecture

Y to X then reconstructed Y CycleGAN Architecture

Loss Functions

Like other GAN, CycleGANs also have a training objective function, which needs to minimize in training process. The loss function is a weighted sum of the following losses.

• Adversarial loss.
• Cycle consistency loss.

Adversarial loss :

It is a loss between the image from the real distribution domain A or domain B, and the images generated by the Generator networks. We have two mapping functions and we will be applying the adversarial loss to both of the mappings.

The adversarial loss for the mapping is

• Where, X is an image from the distribution of domain A, and Y is an image from the distribution of domain B.
• The Discriminator D(Y) try to find the difference between the generated image by the G mapping G(X), and the real image Y from a different distribution of domain B.
• The Discriminator D(X) try to find the difference between the generated image by the F mapping F(Y) and the real image X from the distribution of domain A.
• The objective of G is to minimize the adversarial loss function against a Discriminator D, which constantly tries to maximize it.

Cycle consistency loss:

A cycle consistent mapping function is a function that can translate an image x from domain A to another image y in domain B, and generate back the original image.

• A forward cycle consistent mapping function appears as follows

forward cycle consistent loss

• A backward cycle consistent mapping function looks as follows

backward cycle consistent loss

• The formula for cycle consistency loss is as follows

• With cycle consistency loss, the images reconstructed by F(G(x)) and G(F(y)) will be similar to x and y, respectively.

Full Objective Loss Functions

The complete objective function is a weighted sum of both the adversarial loss and the cycle consistency loss, which is represented as follows

Here, the first adversarial loss as follows

and second adversarial loss as follows

The first adversarial loss is calculated on the Generator A, and the Discriminator B. The second adversarial loss is calculated on the Generator B, and the Discriminator A.

Accompanied jupyter notebook for this post can be found here.

Conclusion

One of the more magical applications of CycleGANs is style transfer. here are list some really cool application of CycleGANs

• Summer scenes converted into wintry ones, and vice versa.

• Monet paintings converted into photo-realistic images.

• Baskets of apples converted into baskets of oranges, and vice versa.

• A photo of a horse converted into a zebra , and vice versa.

• A video of a horse running in a field converted into a zebra.

• Real photo to an artist’s painting.

• A video of a drive taken during the day converted into a nighttime one.
• Flat, low-quality photos converted into what appear to be ones captured by high-end single-lens reflex cameras.
• Satellite image to map.
• Face image to emoji, caricature or anime.
• Body image to the avatar.
• Colorization of grayscale photos.
• Medical scan to a real photo.
• Real photo to an artist’s painting.

I hope this article helped you get started building your own CycleGANs. I think it will at least provides a good explanation and understanding about CycleGANs.