Welcome to `pystiche` ‘s documentation!¶

Note

Click here to download the full example code

Neural Style Transfer with `pystiche`¶

This example showcases how a basic Neural Style Transfer (NST), i.e. image optimization, could be performed with pystiche.

Note

Setup¶

We start this example by importing everything we need and setting the device we will be working on.

 import pystiche
 from pystiche import demo, enc, loss, optim
 from pystiche.image import show_image
 from pystiche.misc import get_device, get_input_image

 print(f"I'm working with pystiche=={pystiche.__version__}")

 device = get_device()
 print(f"I'm working with {device}")

Multi-layer Encoder¶

The content_loss and the style_loss operate on the encodings of an image rather than on the image itself. These encodings are generated by a pretrained encoder. Since we will be using encodings from multiple layers we load a multi-layer encoder. In this example we use the vgg19_multi_layer_encoder() that is based on the VGG19 architecture introduced by Simonyan and Zisserman [SZ14] .

 multi_layer_encoder = enc.vgg19_multi_layer_encoder()
 print(multi_layer_encoder)

Perceptual Loss¶

The core components of every NST are the content_loss and the style_loss. Combined they make up the perceptual loss, i.e. the optimization criterion.

In this example we use the FeatureReconstructionLoss introduced by Mahendran and Vedaldi [MV15] as content_loss. We first extract the content_encoder that generates encodings from the content_layer. Together with the content_weight we can construct the content_loss.

 content_layer = "relu4_2"
 content_encoder = multi_layer_encoder.extract_encoder(content_layer)
 content_weight = 1e0
 content_loss = loss.FeatureReconstructionLoss(
     content_encoder, score_weight=content_weight
 )
 print(content_loss)

We use the GramLoss introduced by Gatys, Ecker, and Bethge [GEB16] as style_loss. Unlike before, we use multiple style_layers. The individual losses can be conveniently bundled in a MultiLayerEncodingLoss.

 style_layers = ("relu1_1", "relu2_1", "relu3_1", "relu4_1", "relu5_1")
 style_weight = 1e3


 def get_style_op(encoder, layer_weight):
     return loss.GramLoss(encoder, score_weight=layer_weight)


 style_loss = loss.MultiLayerEncodingLoss(
     multi_layer_encoder, style_layers, get_style_op, score_weight=style_weight,
 )
 print(style_loss)

We combine the content_loss and style_loss into a joined PerceptualLoss, which will serve as optimization criterion.

 perceptual_loss = loss.PerceptualLoss(content_loss, style_loss).to(device)
 print(perceptual_loss)

Images¶

We now load and show the images that will be used in the NST. The images will be resized to size=500 pixels.

 images = demo.images()
 images.download()
 size = 500

Note

ìmages.download() downloads all demo images upfront. If you only want to download the images for this example remove this line. They will be downloaded at runtime instead.

Note

If you want to work with other images you can load them with read_image():

from pystiche.image import read_image

my_image = read_image("my_image.jpg", size=size, device=device)

 content_image = images["bird1"].read(size=size, device=device)
 show_image(content_image, title="Content image")

 style_image = images["paint"].read(size=size, device=device)
 show_image(style_image, title="Style image")

Neural Style Transfer¶

After loading the images they need to be set as targets for the optimization criterion.

 perceptual_loss.set_content_image(content_image)
 perceptual_loss.set_style_image(style_image)

As a last preliminary step we create the input image. We start from the content_image since this way the NST converges quickly.

 starting_point = "content"
 input_image = get_input_image(starting_point, content_image=content_image)
 show_image(input_image, title="Input image")

Note

If you want to start from a white noise image instead use starting_point = "random" instead:

starting_point = "random"
input_image = get_input_image(starting_point, content_image=content_image)

Finally we run the NST with the image_optimization() for num_steps=500 steps.

In every step the perceptual_loss is calculated nd propagated backward to the pixels of the input_image. If get_optimizer is not specified, as is the case here, the default_image_optimizer(), i.e. LBFGS is used.

 output_image = optim.image_optimization(input_image, perceptual_loss, num_steps=500)

After the NST is complete we show the result.

 show_image(output_image, title="Output image")

Conclusion¶

If you started with the basic NST example without pystiche this example hopefully convinced you that pystiche is a helpful tool. But this was just the beginning: to unleash its full potential head over to the more advanced examples.

Total running time of the script: ( 0 minutes 0.000 seconds)

Estimated memory usage: 0 MB

Note

Click here to download the full example code

Multi-layer Encoder¶

This example showcases the pystiche.enc.MultiLayerEncoder.

We start this example by importing everything we need.

 import itertools
 import time
 from collections import OrderedDict
 from math import floor, log10

 import torch
 from torch import nn
 from torchvision import models

 import pystiche
 from pystiche import enc

 print(f"I'm working with pystiche=={pystiche.__version__}")

In a second preliminary step we define some helper functions to ease the performance analysis later on.

 SI_PREFIXES = {0: "", -3: "m", -6: "µ"}


 def timeit(fn, times=10, cleanup=None):
     total = 0.0
     for _ in range(times):
         start = time.time()
         fn()
         stop = time.time()
         total += stop - start
         if cleanup:
             cleanup()
     return total / times


 def feng(num, unit, digits=3):
     exp = int(floor(log10(num)))
     exp -= exp % 3
     sig = num * 10 ** -exp
     prec = digits - len(str(int(sig)))
     return f"{sig:.{prec}f} {SI_PREFIXES[exp]}{unit}"


 def fsecs(seconds):
     return feng(seconds, "s")


 def ftimeit(fn, msg="The execution took {seconds}.", **kwargs):
     return msg.format(seconds=fsecs(timeit(fn, **kwargs)))


 def fdifftimeit(seq_fn, mle_fn, **kwargs):
     time_seq = timeit(seq_fn, **kwargs)
     time_mle = timeit(mle_fn, **kwargs)

     abs_diff = time_mle - time_seq
     rel_diff = abs_diff / time_seq

     if abs_diff >= 0:
         return (
             f"Encoding the input with the enc.MultiLayerEncoder was "
             f"{fsecs(abs_diff)} ({rel_diff:.0%}) slower."
         )
     else:
         return "\n".join(
             (
                 "Due to the very rough timing method used here, ",
                 "we detected a case where the encoding with the enc.MultiLayerEncoder ",
                 "was actually faster than the boiler-plate nn.Sequential. ",
                 "Since the enc.MultiLayerEncoder has some overhead, ",
                 "this is a measuring error. ",
                 "Still, this serves as indicator that the overhead is small enough, ",
                 "to be well in the measuring tolerance.",
             )
         )

Next up, we define the device we will be testing on as well as the input dimensions.

Note

We encourage the user to play with these parameters and see how the results change. In order to do that, you can use the download buttons at the bottom of this page.

 device = torch.device("cpu")

 batch_size = 32
 num_channels = 3
 height = width = 512

 input = torch.rand((batch_size, num_channels, height, width), device=device)

As a toy example to showcase the MultiLayerEncoder capabilities, we will use a CNN with three layers.

 conv = nn.Conv2d(num_channels, num_channels, 3, padding=1)
 relu = nn.ReLU(inplace=False)
 pool = nn.MaxPool2d(2)

 modules = [("conv", conv), ("relu", relu), ("pool", pool)]

 seq = nn.Sequential(OrderedDict(modules)).to(device)
 mle = enc.MultiLayerEncoder(modules).to(device)
 print(mle)

Before we dive into the additional functionalities of the MultiLayerEncoder we perform a smoke test and assert that it indeed does the same as an torch.nn.Sequential with the same layers.

 assert torch.allclose(mle(input), seq(input))
 print(fdifftimeit(lambda: seq(input), lambda: mle(input)))

As we saw, the MultiLayerEncoder produces the same output as an torch.nn.Sequential but is slower. In the following we will learn what other functionalities a MultiLayerEncoder has to offer that justify this overhead.

Intermediate feature maps¶

By calling the multi-layer encoder with a layer name in addition to the input, the intermediate layers of the MultiLayerEncoder can be accessed. This is helpful if one needs the feature maps from different layers of a model, as is often the case during an NST.

 assert torch.allclose(mle(input, "conv"), conv(input))
 assert torch.allclose(mle(input, "relu"), relu(conv(input)))
 assert torch.allclose(mle(input, "pool"), pool(relu(conv(input))))

For convenience, one can extract a pystiche.enc.SingleLayerEncoder as an interface to the multi-layer encoder for a specific layer.

 sle = mle.extract_encoder("conv")
 assert torch.allclose(sle(input), conv(input))

Caching¶

If the access intermediate feature maps is necessary, as is usually the case in an NST, it is important to only compute every layer once.

A MultiLayerEncoder() enables this functionality by caching already computed feature maps. Thus, after an input is cached, retrieving it is a constant time lookup

In order to enable caching for a layer, it has to be registered first.

Note

The internal cache will be automatically cleared during the backward pass. Since we don’t perform that here, we need to clear it manually by calling clear_cache()

Note

extract_encoder() automatically registers the layer for caching.

 shallow_layers = ("conv", "relu")
 for layer in shallow_layers:
     mle.register_layer(layer)

 mle(input)

 for layer in shallow_layers:
     print(
         ftimeit(
             lambda: mle(input, layer),
             (
                 f"After the forward pass was completed once for the input, "
                 f"extracting the encoding of the intermediate layer '{layer}' "
                 f"took {{seconds}}."
             ),
         )
     )

 mle.clear_cache()

Due to this caching, it doesn’t matter in which order the feature maps are requested:

If a shallow layer is requested before a deeper one, the encoding is later resumed from the feature map of the shallow layer.
If a deep layer is requested before a more shallow one, the feature map of the shallow one is cached while computing the deep layer.

 def fn(layers):
     for layer in layers:
         mle(input, layer)


 for permutation in itertools.permutations(("conv", "relu", "pool")):
     order = f"""'{"' -> '".join(permutation)}'"""
     print(
         ftimeit(
             lambda: fn(permutation),
             f"The encoding of layers {order} took {{seconds}}.",
             cleanup=mle.clear_cache,
         )
     )

Real-world example¶

Up to this point we used a toy example to demonstrate the capabilities of a MultiLayerEncoder. In addition to the boiler-plate MultiLayerEncoder, pystiche has builtin implementations of some well-known CNN architectures that are commonly used in NST papers.

Note

By default, vgg19_multi_layer_encoder() loads weights provided by torchvision. We disable this here since we load the randomly initilaized weights of the torchvision model to enable a comparison.

Note

By default, vgg19_multi_layer_encoder() adds an internal_preprocessing so that the user can simply pass the image as is, without worrying about it. We disable this here to enable a comparison.

Note

By default, vgg19_multi_layer_encoder() disallows in-place operations since after they are carried out, the previous encoding is no longer accessible. In order to enable a fair performance comparison, we allow them here, since they are also used in vgg19().

Note

The fully connected stage of the original VGG19 architecture requires the input to be exactly 224 pixels wide and high [SZ14]. Since this requirement can usually not be met in an NST, the builtin multi-layer encoder only comprises the size invariant convolutional stage. Thus, we only use vgg19().features to enable a comparison.

 seq = models.vgg19()
 mle = enc.vgg19_multi_layer_encoder(
     pretrained=False, internal_preprocessing=False, allow_inplace=True
 )
 mle.load_state_dict(seq.state_dict())

 input = torch.rand((4, 3, 256, 256), device=device)

 assert torch.allclose(mle(input), seq.features(input))
 print(fdifftimeit(lambda: seq.features(input), lambda: mle(input)))

Total running time of the script: ( 0 minutes 0.000 seconds)

Estimated memory usage: 0 MB

Advanced¶

Note

Click here to download the full example code

Guided image optimization¶

This example showcases how a guided, i.e. regionally constraint, NST can be performed in pystiche.

Usually, the style_loss discards spatial information since the style elements should be able to be synthesized regardless of their position in the style_image. Especially for images with clear separated regions style elements might leak into regions where they fit well with respect to the perceptual loss, but don’t belong for a human observer. This can be overcome with spatial constraints also called guides ([GEB+17]).

We start this example by importing everything we need and setting the device we will be working on.

 import pystiche
 from pystiche import demo, enc, loss, optim
 from pystiche.image import guides_to_segmentation, show_image
 from pystiche.misc import get_device, get_input_image

 print(f"I'm working with pystiche=={pystiche.__version__}")

 device = get_device()
 print(f"I'm working with {device}")

In a first step we load and show the images that will be used in the NST.

 images = demo.images()
 images.download()
 size = 500

 content_image = images["castle"].read(size=size, device=device)
 show_image(content_image)

 style_image = images["church"].read(size=size, device=device)
 show_image(style_image)

Unguided image optimization¶

As a baseline we use a default NST with a FeatureReconstructionLoss as content_loss and GramLoss as style_loss.

 multi_layer_encoder = enc.vgg19_multi_layer_encoder()

 content_layer = "relu4_2"
 content_encoder = multi_layer_encoder.extract_encoder(content_layer)
 content_weight = 1e0
 content_loss = loss.FeatureReconstructionLoss(
     content_encoder, score_weight=content_weight
 )

 style_layers = ("relu1_1", "relu2_1", "relu3_1", "relu4_1", "relu5_1")
 style_weight = 1e4


 def get_style_op(encoder, layer_weight):
     return loss.GramLoss(encoder, score_weight=layer_weight)


 style_loss = loss.MultiLayerEncodingLoss(
     multi_layer_encoder, style_layers, get_style_op, score_weight=style_weight,
 )


 perceptual_loss = loss.PerceptualLoss(content_loss, style_loss).to(device)
 print(perceptual_loss)

We set the target images for the optimization criterion.

 perceptual_loss.set_content_image(content_image)
 perceptual_loss.set_style_image(style_image)

We perform the unguided NST and show the result.

 starting_point = "content"
 input_image = get_input_image(starting_point, content_image=content_image)

 output_image = optim.image_optimization(input_image, perceptual_loss, num_steps=500)

 show_image(output_image)

While the result is not completely unreasonable, the building has a strong blueish cast that looks unnatural. Since the optimization was unconstrained the color of the sky was used for the building. In the remainder of this example we will solve this by dividing the images in multiple separate regions.

Guided image optimization¶

For both the content_image and style_image we load regional guides and show them.

Note

In pystiche a guide is a binary image in which the white pixels make up the region that is guided. Multiple guides can be combined into a segmentation for a better overview. In a segmentation the regions are separated by color. You can use guides_to_segmentation() and segmentation_to_guides() to convert one format to the other.

Note

The guides used within this example were created manually. It is possible to generate them automatically [CZP+18], but this is outside the scope of pystiche.

 content_guides = images["castle"].guides.read(size=size, device=device)
 content_segmentation = guides_to_segmentation(content_guides)
 show_image(content_segmentation, title="Content segmentation")

 style_guides = images["church"].guides.read(size=size, device=device)
 style_segmentation = guides_to_segmentation(style_guides)
 show_image(style_segmentation, title="Style segmentation")

The content_image is separated in three regions: the "building", the "sky", and the "water".

Note

Since no water is present in the style image we reuse the "sky" for the "water" region.

 regions = ("building", "sky", "water")

 style_guides["water"] = style_guides["sky"]

Since the stylization should be performed for each region individually, we also need separate losses. Within each region we use the same setup as before. Similar to how a MultiLayerEncodingLoss bundles multiple operators acting on different layers a MultiRegionLoss bundles multiple losses acting in different regions.

The guiding is only needed for the style_loss since the content_loss by definition honors the position of the content during the optimization. Thus, the previously defined content_loss is combined with the new regional style_loss.

 def get_region_op(region, region_weight):
     return loss.MultiLayerEncodingLoss(
         multi_layer_encoder, style_layers, get_style_op, score_weight=region_weight,
     )


 style_loss = loss.MultiRegionLoss(regions, get_region_op, score_weight=style_weight)

 perceptual_loss = loss.PerceptualLoss(content_loss, style_loss).to(device)
 print(perceptual_loss)

The content_loss is unguided and thus the content image can be set as we did before. For the style_loss we use the same style_image for all regions and only vary the guides.

 perceptual_loss.set_content_image(content_image)

 for region in regions:
     perceptual_loss.set_style_image(
         style_image, guide=style_guides[region], region=region
     )
     perceptual_loss.set_content_guide(content_guides[region], region=region)

We rerun the optimization with the new constrained optimization criterion and show the result.

 starting_point = "content"
 input_image = get_input_image(starting_point, content_image=content_image)

 output_image = optim.image_optimization(input_image, perceptual_loss, num_steps=500)

 show_image(output_image)

With regional constraints we successfully removed the blueish cast from the building which leads to an overall higher quality. Unfortunately, reusing the sky region for the water did not work out too well: due to the vibrant color, the water looks unnatural.

Fortunately, this has an easy solution. Since we are already using separate losses for each region we are not bound to use only a single style_image: if required, we can use a different style_image for each region.

Guided image optimization with multiple styles¶

We load a second style image that has water in it.

 second_style_image = images["cliff"].read(size=size, device=device)
 show_image(second_style_image, "Second style image")

 second_style_guides = images["cliff"].guides.read(size=size, device=device)
 show_image(guides_to_segmentation(second_style_guides), "Second style segmentation")

We can reuse the previously defined criterion and only change the style_image and style_guides in the "water" region.

 region = "water"
 perceptual_loss.set_style_image(
     second_style_image, guide=second_style_guides[region], region=region
 )

Finally, we rerun the optimization again with the new constraints.

 starting_point = "content"
 input_image = get_input_image(starting_point, content_image=content_image)

 output_image = optim.image_optimization(input_image, perceptual_loss, num_steps=500)

 show_image(output_image)

Compared to the two previous results we now achieved the highest quality. Nevertheless, This approach has its downsides : since we are working with multiple images in multiple distinct regions, the memory requirement is higher compared to the other approaches. Furthermore, compared to the unguided NST, the guides have to be provided together with the for the content and style images.

Total running time of the script: ( 0 minutes 0.000 seconds)

Estimated memory usage: 0 MB

Note

Click here to download the full example code

Image optimization with pyramid¶

This example showcases how an image pyramid is integrated in an NST with pystiche.

With an image pyramid the optimization is not performed on a single but rather on multiple increasing resolutions. This procedure is often dubbed coarse-to-fine, since on the lower resolutions coarse structures are synthesized whereas on the higher levels the details are carved out.

This technique has the potential to reduce the convergence time as well as to enhance the overall result [LW16][GEB+17].

We start this example by importing everything we need and setting the device we will be working on.

 import time

 import pystiche
 from pystiche import demo, enc, loss, optim, pyramid
 from pystiche.image import show_image
 from pystiche.misc import get_device, get_input_image

 print(f"I'm working with pystiche=={pystiche.__version__}")

 device = get_device()
 print(f"I'm working with {device}")

At first we define a PerceptualLoss that is used as optimization criterion.

 multi_layer_encoder = enc.vgg19_multi_layer_encoder()


 content_layer = "relu4_2"
 content_encoder = multi_layer_encoder.extract_encoder(content_layer)
 content_weight = 1e0
 content_loss = loss.FeatureReconstructionLoss(
     content_encoder, score_weight=content_weight
 )


 style_layers = ("relu3_1", "relu4_1")
 style_weight = 2e0


 def get_style_op(encoder, layer_weight):
     return loss.MRFLoss(encoder, patch_size=3, stride=2, score_weight=layer_weight)


 style_loss = loss.MultiLayerEncodingLoss(
     multi_layer_encoder, style_layers, get_style_op, score_weight=style_weight,
 )

 perceptual_loss = loss.PerceptualLoss(content_loss, style_loss).to(device)
 print(perceptual_loss)

Next up, we load and show the images that will be used in the NST.

 images = demo.images()
 images.download()
 size = 500

 content_image = images["bird2"].read(size=size, device=device)
 show_image(content_image, title="Content image")
 perceptual_loss.set_content_image(content_image)

 style_image = images["mosaic"].read(size=size, device=device)
 show_image(style_image, title="Style image")
 perceptual_loss.set_style_image(style_image)

Image optimization without pyramid¶

As a baseline we use a standard image optimization without pyramid.

 starting_point = "content"
 input_image = get_input_image(starting_point, content_image=content_image)
 show_image(input_image, title="Input image")

We time the NST performed by image_optimization() and show the result.

 start_without_pyramid = time.time()
 output_image = optim.image_optimization(input_image, perceptual_loss, num_steps=400)
 stop_without_pyramid = time.time()

 show_image(output_image, title="Output image without pyramid")

 elapsed_time_without_pyramid = stop_without_pyramid - start_without_pyramid
 print(
     f"Without pyramid the optimization took {elapsed_time_without_pyramid:.0f} seconds."
 )

As you can see the small blurry branches on the left side of the image were picked up by the style transfer. They distort the mosaic pattern, which minders the quality of the result. In the next section we tackle this by focusing on coarse elements first and add the details afterwards.

Image optimization with pyramid¶

Opposed to the prior examples we now want to perform an NST on multiple resolutions. In pystiche this handled by an ImagePyramid . The resolutions are selected by specifying the edge_sizes of the images on each level . The optimization is performed for num_steps on the different levels.

The resizing of all images, i.e. input_image and target images (content_image and style_image) is handled by the pyramid. For that we need to register the perceptual loss (criterion) as one of the resize_targets.

Note

By default the edge_sizes correspond to the shorter edge of the images. To change that you can pass edge="long". For fine-grained control you can also pass a sequence comprising "short" and "long" to select the edge for each level separately. Its length has to match the length of edge_sizes.

Note

For a fine-grained control over the number of steps on each level you can pass a sequence to select the num_steps for each level separately. Its length has to match the length of edge_sizes.

 edge_sizes = (250, 500)
 num_steps = 200
 image_pyramid = pyramid.ImagePyramid(
     edge_sizes, num_steps, resize_targets=(perceptual_loss,)
 )
 print(image_pyramid)

With a pyramid the NST is performed by pyramid_image_optimization(). We time the execution and show the result afterwards.

Note

We regenerate the input_image since it was changed inplace during the first optimization.

 input_image = get_input_image(starting_point, content_image=content_image)

 start_with_pyramid = time.time()
 output_image = optim.pyramid_image_optimization(
     input_image, perceptual_loss, image_pyramid
 )
 stop_with_pyramid = time.time()

 show_image(output_image, title="Output image with pyramid")

 elapsed_time_with_pyramid = stop_with_pyramid - start_with_pyramid
 relative_decrease = 1.0 - elapsed_time_with_pyramid / elapsed_time_without_pyramid
 print(
     f"With pyramid the optimization took {elapsed_time_with_pyramid:.0f} seconds. "
     f"This is a {relative_decrease:.0%} decrease."
 )

With the coarse-to-fine architecture of the image pyramid, the stylization of the blurry background branches is reduced leaving the mosaic pattern mostly intact. On top of this quality improvement the execution time is significantly lower while performing the same number of steps.

Total running time of the script: ( 0 minutes 0.000 seconds)

Estimated memory usage: 0 MB

Note

Click here to download the full example code

Model optimization¶

This example showcases how an NST based on model optimization can be performed in pystiche. It closely follows the official PyTorch example which in turn is based on [JAL16].

We start this example by importing everything we need and setting the device we will be working on.

 import contextlib
 import os
 import time
 from collections import OrderedDict
 from os import path

 import torch
 from torch import hub, nn
 from torch.nn.functional import interpolate
 from torch.utils.data import DataLoader
 from torchvision import transforms

 import pystiche
 from pystiche import demo, enc, loss, optim
 from pystiche.data import ImageFolderDataset
 from pystiche.image import show_image
 from pystiche.misc import get_device

 print(f"I'm working with pystiche=={pystiche.__version__}")

 device = get_device()
 print(f"I'm working with {device}")

Transformer¶

In contrast to image optimization, for model optimization we need to define a transformer that, after it is trained, performs the stylization. In general different architectures are possible ([JAL16][ULVL16]). For this example we use an encoder-decoder architecture.

Before we define the transformer, we create some helper modules to reduce the clutter.

In the decoder we need to upsample the image. While it is possible to achieve this with a ConvTranspose2d, it was found that traditional upsampling followed by a standard convolution produces fewer artifacts. Thus, we create an module that wraps torch.nn.functional.interpolate().

 class Interpolate(nn.Module):
     def __init__(self, scale_factor=1.0, mode="nearest"):
         super().__init__()
         self.scale_factor = scale_factor
         self.mode = mode

     def forward(self, input):
         return interpolate(input, scale_factor=self.scale_factor, mode=self.mode,)

     def extra_repr(self):
         extras = []
         if self.scale_factor:
             extras.append(f"scale_factor={self.scale_factor}")
         if self.mode != "nearest":
             extras.append(f"mode={self.mode}")
         return ", ".join(extras)

For the transformer architecture we will be using, we need to define a convolution module with some additional capabilities. In particular, it needs to be able to

optionally upsample the input,
pad the input in order for the convolution to be size-preserving,
optionally normalize the output, and
optionally pass the output through an activation function.

Note

Instead of BatchNorm2d we use InstanceNorm2d to normalize the output since it gives better results for NST [UVL16].

 class Conv(nn.Module):
     def __init__(
         self,
         in_channels,
         out_channels,
         kernel_size,
         stride=1,
         upsample=False,
         norm=True,
         activation=True,
     ):
         super().__init__()
         self.upsample = Interpolate(scale_factor=stride) if upsample else None
         self.pad = nn.ReflectionPad2d(kernel_size // 2)
         self.conv = nn.Conv2d(
             in_channels, out_channels, kernel_size, stride=1 if upsample else stride
         )
         self.norm = nn.InstanceNorm2d(out_channels, affine=True) if norm else None
         self.activation = nn.ReLU() if activation else None

     def forward(self, input):
         if self.upsample:
             input = self.upsample(input)

         output = self.conv(self.pad(input))

         if self.norm:
             output = self.norm(output)
         if self.activation:
             output = self.activation(output)

         return output

It is common practice to append a few residual blocks after the initial convolutions to the encoder to enable it to learn more descriptive features.

 class Residual(nn.Module):
     def __init__(self, channels):
         super().__init__()
         self.conv1 = Conv(channels, channels, kernel_size=3)
         self.conv2 = Conv(channels, channels, kernel_size=3, activation=False)

     def forward(self, input):
         output = self.conv2(self.conv1(input))
         return output + input

It can be useful for the training to transform the input into another value range, for example from $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \closedinterval{0}{1}$ to $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \closedinterval{0}{255}$.

 class FloatToUint8Range(nn.Module):
     def forward(self, input):
         return input * 255.0


 class Uint8ToFloatRange(nn.Module):
     def forward(self, input):
         return input / 255.0

Finally, we can put all pieces together.

Note

You can access this transformer through pystiche.demo.transformer().

 class Transformer(nn.Module):
     def __init__(self):
         super().__init__()
         self.encoder = nn.Sequential(
             Conv(3, 32, kernel_size=9),
             Conv(32, 64, kernel_size=3, stride=2),
             Conv(64, 128, kernel_size=3, stride=2),
             Residual(128),
             Residual(128),
             Residual(128),
             Residual(128),
             Residual(128),
         )
         self.decoder = nn.Sequential(
             Conv(128, 64, kernel_size=3, stride=2, upsample=True),
             Conv(64, 32, kernel_size=3, stride=2, upsample=True),
             Conv(32, 3, kernel_size=9, norm=False, activation=False),
         )

         self.preprocessor = FloatToUint8Range()
         self.postprocessor = Uint8ToFloatRange()

     def forward(self, input):
         input = self.preprocessor(input)
         output = self.decoder(self.encoder(input))
         return self.postprocessor(output)


 transformer = Transformer().to(device)
 print(transformer)

Perceptual loss¶

Although model optimization is a different paradigm, the perceptual_loss is the same as for image optimization.

Note

In some implementations, such as the PyTorch example and [JAL16], one can observe that the gram_matrix(), used as style representation, is not only normalized by the height and width of the feature map, but also by the number of channels. If used together with a mse_loss(), the normalization is performed twice. While this is unintended, it affects the training. In order to keep the other hyper parameters on par with the PyTorch example, we also adopt this change here.

 multi_layer_encoder = enc.vgg16_multi_layer_encoder()

 content_layer = "relu2_2"
 content_encoder = multi_layer_encoder.extract_encoder(content_layer)
 content_weight = 1e5
 content_loss = loss.FeatureReconstructionLoss(
     content_encoder, score_weight=content_weight
 )


 class GramOperator(loss.GramLoss):
     def enc_to_repr(self, enc: torch.Tensor) -> torch.Tensor:
         repr = super().enc_to_repr(enc)
         num_channels = repr.size()[1]
         return repr / num_channels


 style_layers = ("relu1_2", "relu2_2", "relu3_3", "relu4_3")
 style_weight = 1e10
 style_loss = loss.MultiLayerEncodingLoss(
     multi_layer_encoder,
     style_layers,
     lambda encoder, layer_weight: GramOperator(encoder, score_weight=layer_weight),
     layer_weights="sum",
     score_weight=style_weight,
 )

 perceptual_loss = loss.PerceptualLoss(content_loss, style_loss)
 perceptual_loss = perceptual_loss.to(device)
 print(perceptual_loss)

Training¶

In a first step we load the style image that will be used to train the transformer.

 images = demo.images()
 size = 500

 style_image = images["paint"].read(size=size, device=device)
 show_image(style_image)

The training of the transformer is performed similar to other models in PyTorch. In every optimization step a batch of content images is drawn from a dataset, which serve as input for the transformer as well as content_image for the perceptual_loss. While the style_image only has to be set once, the content_image has to be reset in every iteration step.

While this can be done with a boilerplate optimization loop, pystiche provides multi_epoch_model_optimization() that handles the above for you.

Note

If the perceptual_loss is a PerceptualLoss, as is the case here, the update of the content_image is performed automatically. If that is not the case or you need more complex update behavior, you need to specify a criterion_update_fn.

Note

If you do not specify an optimizer, the default_model_optimizer(), i.e. Adam is used.

 def train(*, transformer, root, batch_size, epochs, image_size):
     if root is None:
         raise RuntimeError("You forgot to define a root image directory.")

     transform = nn.Sequential(
         transforms.Resize(image_size), transforms.CenterCrop(image_size),
     )
     dataset = ImageFolderDataset(root, transform=transform)
     image_loader = DataLoader(dataset, batch_size=batch_size)

     perceptual_loss.set_style_image(style_image)

     return optim.multi_epoch_model_optimization(
         image_loader, transformer.train(), perceptual_loss, epochs=epochs,
     )

Depending on the dataset and your setup the training can take a couple of hours. To avoid this, we provide transformer weights that were trained with the scheme above.

 def download():
     # Unfortunately, torch.hub.load_state_dict_from_url has no option to disable
     # printing the downloading process. Since this would clutter the output, we
     # suppress it completely.
     @contextlib.contextmanager
     def suppress_output():
         with open(os.devnull, "w") as devnull:
             with contextlib.redirect_stdout(devnull), contextlib.redirect_stderr(
                 devnull
             ):
                 yield

     with suppress_output():
         return hub.load_state_dict_from_url(
             "https://download.pystiche.org/models/example_transformer.pth"
         )

Note

The weights of the provided transformer were trained with the 2014 training images of the COCO dataset. The training was performed for num_epochs=2 and batch_size=4. Each image was center-cropped to 256 x 256 pixels.

Note

If you want to perform the training yourself, set root to a location of a folder of images.

 root = None
 checkpoint = "example_transformer.pth"

 if root is None:
     state_dict = torch.load(checkpoint) if path.exists(checkpoint) else download()
     transformer.load_state_dict(state_dict)
 else:
     transformer = train(
         transformer=transformer, root=root, batch_size=4, epochs=2, image_size=256,
     )
     state_dict = OrderedDict(
         [
             (name, parameter.detach().cpu())
             for name, parameter in transformer.state_dict().items()
         ]
     )
     torch.save(state_dict, checkpoint)

Neural Style Transfer¶

In order to perform the NST, we load an image we want to stylize.

 input_image = images["bird1"].read(size=size, device=device)
 show_image(input_image)

After the transformer is trained we can now perform an NST with a single forward pass. To do this, the transformer is simply called with the input_image.

 transformer.eval()

 start = time.time()

 with torch.no_grad():
     output_image = transformer(input_image)

 stop = time.time()

 show_image(output_image, title="Output image")

Compared to NST via image optimization, the stylization is performed multiple orders of magnitudes faster. Given capable hardware, NST via model optimization enables real-time stylization for example of a video feed.

 print(f"The stylization took {(stop - start) * 1e3:.0f} milliseconds.")

Total running time of the script: ( 0 minutes 0.000 seconds)

Estimated memory usage: 0 MB

Command-line interface¶

For simple tasks pystiche provides a command-line interface (CLI). For example, a simple NST with the builtin demo images can be performed with:

$pystiche bird1 paint

In version

$pystiche --version
1.1.0.dev50+g71217c2

the CLI looks like this:

$pystiche --help
usage: pystiche [-h] [-V] [-v] [-o OUTPUT_IMAGE] [-n NUM_STEPS] [-d DEVICE]
                [-s STARTING_POINT]
                [--multi-layer-encoder MULTI_LAYER_ENCODER]
                [--content-loss CONTENT_LOSS]
                [--content-layers CONTENT_LAYERS]
                [--content-size CONTENT_SIZE]
                [--content-weight CONTENT_WEIGHT] [--style-loss STYLE_LOSS]
                [--style-layers STYLE_LAYERS] [--style-size STYLE_SIZE]
                [--style-weight STYLE_WEIGHT]
                content_image style_image

Performs simple Neural Style Transfers (NSTs) with image optimization. For
more complex tasks, have a look at the Python API at
https://docs.pystiche.org.

positional arguments:
  content_image         Image containing the content for the style transfer.
                        Can be a path to an image file or the name of a
                        pystiche.demo image.
  style_image           Image containing the style for the style transfer. Can
                        be a path to an image file or the name of a
                        pystiche.demo image.

optional arguments:
  -h, --help            Show this message and exit.
  -V, --version         Show pystiche's version and exit.
  -v, --verbose         Print additional information to STDOUT.
  -o OUTPUT_IMAGE, --output-image OUTPUT_IMAGE
                        Path, the output image will be saved to. If omitted,
                        the output image will be saved to
                        'pystiche_{timestamp}.jpg' in the current directory.
  -n NUM_STEPS, --num-steps NUM_STEPS
                        Number of optimization steps. Defaults to '500'.
  -d DEVICE, --device DEVICE
                        Device, the optimization is performed on. If
                        available, defaults to 'cuda' and falls back to 'cpu'
                        otherwise.
  -s STARTING_POINT, --starting-point STARTING_POINT
                        Starting point of the optimization. Can be 'content'
                        (default) to start from the content image, 'random' to
                        start from a white noise image, a path to an image
                        file, or a name of a pystiche.demo image.
  --multi-layer-encoder MULTI_LAYER_ENCODER, --mle MULTI_LAYER_ENCODER
                        Can be any pretrained multi-layer encoder from
                        pystiche.enc, e.g. 'vgg19' (default) or 'alexnet'.

Content options:
  --content-loss CONTENT_LOSS, --cl CONTENT_LOSS
                        Can be any comparison loss from pystiche.loss, e.g.
                        'FeatureReconstruction' (default).
  --content-layers CONTENT_LAYERS, --cla CONTENT_LAYERS
                        Layers of the MULTI_LAYER_ENCODER used to encode the
                        content representation, e.g. 'relu4_2' (default).
                        Multiple layers can be given as a comma separated
                        list.
  --content-size CONTENT_SIZE, --cs CONTENT_SIZE
                        Size in pixels the CONTENT_IMAGE will be resized to
                        before the optimization, e.g. '500' (default).
  --content-weight CONTENT_WEIGHT, --cw CONTENT_WEIGHT
                        Optimization weight for the CONTENT_LOSS, e.g. '1e0'
                        (default). Higher values lead to more focus on the
                        content.

Style options:
  --style-loss STYLE_LOSS, --sl STYLE_LOSS
                        Can be any comparison loss from pystiche.loss, e.g.
                        'Gram' (default) or 'MRF'.
  --style-layers STYLE_LAYERS, --sla STYLE_LAYERS
                        Layers of the MULTI_LAYER_ENCODER used to encode the
                        content representation. Multiple layers can be given
                        as a comma separated list, e.g.
                        'relu1_1,relu2_1,relu3_1,relu4_1,relu5_1' (default).
  --style-size STYLE_SIZE, --ss STYLE_SIZE
                        Size in pixels the STYLE_IMAGE will be resized to
                        before the optimization, e.g. '500' (default).
  --style-weight STYLE_WEIGHT, --sw STYLE_WEIGHT
                        Optimization weight for the STYLE_LOSS, e.g. '1e3'
                        (default). Higher values lead to more focus on the
                        style.

Package reference¶

`pystiche`¶

pystiche.home()¶

Local directory to save downloaded images and guides. Defaults to ~/.cache/pystiche but can be overwritten with the PYSTICHE_HOME environment variable.

Return type: str

Objects¶

class pystiche.ComplexObject¶

Object with a complex representation. See pystiche.misc.build_complex_obj_repr() for details.

_named_children()¶

Yields: Internal named children.

Note

If subclassed, this method should yield the named children of the superclass alongside yielding the new named children.

Return type: Iterator[Tuple[str, Any]]

_properties()¶

Return type: Dict[str, Any]
Returns: Internal properties.

Note

If subclassed, this method should integrate the new properties in the properties of the superclass.

extra_named_children()¶

Yields: Extra named children.
Return type: Iterator[Tuple[str, Any]]

extra_properties()¶

Return type: Dict[str, Any]
Returns: Extra properties.

named_children()¶

Yields: Internal and extra named children.
Return type: Iterator[Tuple[str, Any]]

properties()¶

Return type: Dict[str, Any]
Returns: Internal and extra properties.

class pystiche.LossDict(losses=())¶

Hierarchic dictionary of scalar torch.Tensor losses. Levels are seperated by "." in the names.

Parameters: losses (Sequence[Tuple[str, Union[Tensor, LossDict]]]) – Optional named losses.

__mul__(other)¶

Multiplies all entries with a scalar.

Parameters: other (SupportsFloat) – Scalar multiplier.
Return type: LossDict

__setitem__(name, loss)¶

Add a named loss to the entries.

Parameters

name (str) – Name of the loss.
loss (Union[Tensor, LossDict]) – If torch.Tensor, it has to be scalar. If LossDict, it is unpacked and the entries are added as level below of name.

Raises

TypeError – If loss is torch.Tensor but isn’t scalar.

Return type

aggregate(max_depth)¶

Aggregate all entries up to a given maximum depth.

Parameters: max_depth (int) – If 0 returns sum of all entries as scalar torch.Tensor.
Return type: Union[Tensor, LossDict]

backward(*args, **kwargs)¶

Computes the gradient of all entries with respect to the graph leaves. See torch.Tensor.backward() for details.

Return type: None

item()¶

Return type: float
Returns: The sum of all entries as standard Python number.

total()¶

Return type: Tensor
Returns: Sum of all entries as scalar tensor.

class pystiche.Module(named_children=None, indexed_children=None)¶

torch.nn.Module with the enhanced representation options of pystiche.ComplexObject.

Parameters

named_children (Optional[Iterable[Tuple[str, Module]]]) – Optional children modules that are added by their names.
indexed_children (Optional[Sequence[Module]]) – Optional children modules that are added with successive indices as names.

Note

named_children and indexed_children are mutually exclusive parameters.

torch_repr()¶

Return type: str
Returns: Native torch representation.

Math¶

pystiche.nonnegsqrt(x)¶

Safely calculates the square-root of a non-negative input

\[\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \begin{split}\fun{nonnegsqrt}{x} = \begin{cases} \sqrt{x} &\quad\text{if } x \ge 0 \\ 0 &\quad\text{otherwise} \end{cases}\end{split}\]

Note

This operation is useful in situations where the input tensor is strictly non-negative from a theoretical standpoint, but might be negative due to numerical instabilities.

Parameters: x (Tensor) – Input tensor.
Return type: Tensor

pystiche.gram_matrix(x, normalize=False)¶

Calculates the channel-wise Gram matrix of a batched input tensor.

Given a tensor $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} x$ of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times C \times N_1 \times \dots \times N_D$ each element of the single-sample Gram matrix $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} G_{b,c_1 c_2}$ with $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} b \in 1,\dots,B$ and $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} c_1,\,c_2 \in 1,\dots,C$ is calculated by

\[\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} G_{b,c_1 c_2} = \dotproduct{\fun{vec}{x_{b, c_1}}}{\fun{vec}{x_{b, c_2}}}\]

where $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \dotproduct{\cdot}{\cdot}$ denotes the dot product and $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \fun{vec}{\cdot}$ denotes the vectorization function .

Parameters

x (Tensor) – Input tensor of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times C \times N_1 \times \dots \times N_D$
normalize (bool) – If True, normalizes the Gram matrix $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} G$ by $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \prod\limits_{d=1}^{D} N_d$ to keep the value range similar for different sized inputs. Defaults to False.

Return type

Returns

Channel-wise Gram matrix G of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times C \times C$.

pystiche.cosine_similarity(x1, x2, eps=1e-08, batched_input=None)¶

Calculates the cosine similarity between the samples of x1 and x2.

Parameters

x1 (Tensor) – First input of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times S_1 \times N_1 \times \dots \times N_D$.
x2 (Tensor) – Second input of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times S_2 \times N_1 \times \dots \times N_D$.
eps (float) – Small value to avoid zero division. Defaults to 1e-8.
batched_input (Optional[bool]) – If False, treat the first dimension of the inputs as sample dimension, i.e. $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} S \times N_1 \times \dots \times N_D$. Defaults to True.

Return type

Returns

Similarity matrix of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times S_1 \times S_2$ in which every element represents the cosine similarity between the corresponding samples $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} S$ of x1 and x2. If batched_input is False, the output shape is $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} S_1 \times S_2$

`pystiche.data`¶

class pystiche.data.LocalImage(file, collect_local_guides=True, guides=None, transform=None, note=None)¶

read(root=None, **read_image_kwargs)¶

Read the image from file with pystiche.image.read_image() and optionally apply transform.

Parameters

root (Optional[str]) – Optional root directory if the file is a relative path.
**read_image_kwargs – Optional parameters passed to pystiche.image.read_image().

Return type

class pystiche.data.LocalImageCollection(images)¶

read(root=None, **read_image_kwargs)¶

Read the images from file.

Parameters

root (Optional[str]) – Optional root directory if the file is a relative path.
**read_image_kwargs – Optional parameters passed to pystiche.image.read_image().

Return type

Dict[str, Tensor]

Returns

Dictionary with the name image pairs.

class pystiche.data.DownloadableImage(url, title=None, author=None, date=None, license=None, md5=None, file=None, guides=None, prefix_guide_files=True, transform=None, note=None)¶

download(root=None, overwrite=False)¶

Download the image and if applicable the guides from their URL. If the correct MD5 checksum is known, it is verified first. If it checks out the file not re-downloaded.

Parameters

root (Optional[str]) – Optional root directory for the download if the file is a relative path. Defaults to pystiche.home().
overwrite (bool) – Overwrites files if they already exists or the MD5 checksum does not match. Defaults to False.

Return type

static generate_file(url, title, author)¶

Generate a filename from the supplied information from the following scheme:

If title and author are None, the ending of url is used.
If one of title or author is not None, it is used as filename where spaces are replaced by underscores.
If title and author are not None, the filename is generated as above separating both path with double underscores.

Parameters

url (str) – URL to the image.
title (Optional[str]) – Optional title of the image.
author (Optional[str]) – Optional author of the image

Return type

str

read(root=None, download=None, overwrite=False, **read_image_kwargs)¶

Read the image from file with pystiche.image.read_image(). If available the transform is applied afterwards.

Parameters

root (Optional[str]) – Optional root directory if the file is a relative path. Defaults to pystiche.home().
download (Optional[bool]) – If True, downloads the image first. Defaults to False if the file already exists and the MD5 checksum is not known. Otherwise defaults to True.
overwrite (bool) – If downloaded, overwrites files if they already exists or the MD5 checksum does not match. Defaults to False.
**read_image_kwargs – Optional parameters passed to pystiche.image.read_image().

Return type

class pystiche.data.DownloadableImageCollection(images)¶

download(root=None, overwrite=False)¶

Download all images and if applicable their guides from their URLs. See pystiche.data.DownloadableImage.download() for details.

Parameters

root (Optional[str]) – Optional root directory for the download if the file is a relative path. Defaults to pystiche.home().
overwrite (bool) – Overwrites files if they already exists or the MD5 checksum does not match. Defaults to False.

Return type

torch.nn.Module.load_state_dict()

read(root=None, download=None, overwrite=False, **read_image_kwargs)¶

Read the images from file. See pystiche.data.DownloadableImage.read() for details.

Parameters

root (Optional[str]) – Optional root directory if the file is a relative path. Defaults to pystiche.home().
download (Optional[bool]) – If True, downloads the image first. Defaults to False if the file already exists and the MD5 checksum is not known. Otherwise defaults to True.
overwrite (bool) – If downloaded, overwrites files if they already exists or the MD5 checksum does not match. Defaults to False.
**read_image_kwargs – Optional parameters passed to pystiche.image.read_image().

Return type

Dict[str, Tensor]

Returns

Dictionary with the name image pairs.

`pystiche.demo`¶

pystiche.demo.images()¶

Collection of images used in the usage examples.

Return type: DownloadableImageCollection

name	image
bird1
bird2
castle
church
cliff
mosaic
paint

pystiche.demo.transformer()¶

Basic transformer for model-based optimization.

The transformer is compatible with the official PyTorch example which in turn is based on [JAL16]

Return type: Module

`pystiche.enc`¶

class pystiche.enc.Encoder(named_children=None, indexed_children=None)¶

ABC for all encoders. Invokes Encoder.forward() if called.

abstract forward(input)¶

Encodes the given input.

Note

This method has to be overwritten in every subclass.

Return type: Tensor

abstract propagate_guide(guide)¶

Encodes the given guide.

Note

This method has to be overwritten in every subclass.

Return type: Tensor

class pystiche.enc.SequentialEncoder(modules)¶

Encoder that operates in sequential manner. Invokes Encoder.forward() if called.

Parameters: modules (Sequence[Module]) – Sequential modules.

class pystiche.enc.MultiLayerEncoder(modules)¶

Sequential encoder with convenient access to intermediate layers.

Parameters: modules (Sequence[Tuple[str, Module]]) – Named modules that serve as basis for the encoding.

registered_layers¶: Layers, on which the encodings will be cached during the forward() pass.

__call__(*args, **kwargs)¶

Invokes forward().

Return type: Any

__contains__(layer)¶

Is the layer part of the multi-layer encoder?

Parameters: layer (str) – Layer to be checked.
Return type: bool

clear_cache()¶

Clear the internal cache.

Return type: None

encode(input, layers)¶

Encode the input on layers.

Parameters

input (Tensor) – Input to be encoded.
layers (Sequence[str]) – Layers on which the input should be encoded.

Return type

Tuple[Tensor, …]

extract_encoder(layer)¶

Extract a SingleLayerEncoder for the layer and register it.

Parameters: layer (str) – Layer.
Return type: SingleLayerEncoder

forward(input, layer=None, cache=None, to_cache=None)¶

Encode the input.

Parameters

input (Tensor) – Input to be encoded.
layer (Optional[str]) – Layer on which the input should be encoded. If omitted, defaults to the last layer in the multi-layer encoder.
cache (Optional[Dict[str, Tensor]]) – Encoding cache. If omitted, defaults to the the internal cache.
to_cache (Optional[Collection[str]]) – Layers, of which the encodings should be cached. If omitted, defaults to registered_layers.

Examples

>>> modules = [("conv", nn.Conv2d(3, 3, 3)), ("pool", nn.MaxPool2d(2))]
>>> mle = pystiche.enc.MultiLayerEncoder(modules)
>>> input = torch.rand(1, 3, 128, 128)
>>> output = mle(input, "conv")

Return type: Tensor

propagate_guide(guide, layers, method='simple', allow_empty=False)¶

Propagate the guide on the given layers.

Parameters

guide (Tensor) – Guide.
layers (Sequence[str]) – Layers.
allow_empty (bool) – If True, allow the propagated guides to become empty. Defaults to False.

Return type

Tuple[Tensor, …]

Returns

Tuple of guides which order corresponds to layers.

register_layer(layer)¶

Parameters: layer (str) – Layer to be registered.
Return type: None

verify(layer)¶

Verifies that a layer is part of the multi-layer encoder.

Parameters: layer (str) – Layer to be checked.
Raises: ValueError – If layer is not part of the multi-layer encoder.
Return type: None

class pystiche.enc.SingleLayerEncoder(multi_layer_encoder, layer)¶

Encoder extracted from a MultiLayerEncoder that operates on a single layer. Invokes SingleLayerEncoder.forward() if called.

multi_layer_encoder¶: Corresponding multi-layer encoder.

layer¶: Encoding layer.

forward(input)¶

Encode the given input on SingleLayerEncoder.layer of SingleLayerEncoder.multi_layer_encoder.

Parameters: input_image – Input image.
Return type: Tensor

propagate_guide(guide)¶

Propagate the given guide on SingleLayerEncoder.layer of SingleLayerEncoder.multi_layer_encoder.

Parameters: guide (Tensor) – Guide.
Return type: Tensor

Models¶

class pystiche.enc.ModelMultiLayerEncoder(pretrained=True, framework='torch', internal_preprocessing=True, allow_inplace=False)¶

Multi-layer encoder based on a pre-defined model.

Parameters

pretrained (bool) – If True, loads builtin weights. Defaults to True.
framework (str) – Name of the framework that was used to train the builtin weights. Defaults to "torch".
internal_preprocessing (bool) – If True, adds a preprocessing layer for the selected framework as first layer. Defaults to True.
allow_inplace (bool) –
If True, allows inplace operations to reduce the memory requirement during the forward pass. Defaults to False.

Warning

After performing an inplace operation the encodings of the previous layer is no longer accessible. Only use this if you are sure that you do not need these encodings.

abstract collect_modules(inplace)¶

Collect modules of a base model with more descriptive names.

Parameters: inplace (bool) – If True, when possible, modules should use inplace operations.
Return type: Tuple[List[Tuple[str, Module]], Dict[str, str]]
Returns: List of name-module-pairs as well as a dictionary mapping the new, more descriptive names to the original ones.

load_state_dict(state_dict, strict=True, map_names=True, framework='unknown')¶

Loads parameters and buffers from the state_dict.

Parameters

state_dict (Dict[str, Tensor]) – State dictionary.
strict (bool) – Enforce matching keys in state_dict and the internal states.
map_names (bool) – If True, maps the names names in state_dict of the underlying model to the more descriptive names generated by collect_modules(). Defaults to True.
framework (str) –
Name of the framework that was used to train the weights in state_dict. Defaults to "unknown".

Note

This has no effect on the behavior, but makes the representation of the ModelMultiLayerEncoder more descriptive.

Return type

_IncompatibleKeys

Returns

Named tuple with missing_keys and unexpected_keys fields.

See also

load_state_dict_from_url(framework, strict=True, map_names=True, check_hash=True, **kwargs)¶

Downloads and loads parameters and buffers trained with framework.

Parameters

framework (str) – Name of the framework that was used to train the weights of the state_dict.
strict (bool) – Enforce matching keys in state_dict and the internal states.
map_names (bool) – If True, maps the names names in state_dict of the underlying model to the more descriptive names generated by collect_modules(). Defaults to True.
check_hash (bool) – If True, checks if the hash postfix of the URL matches the SHA256 hash of the downloaded state_dict. Defaults to True.
kwargs (Any) – Optional arguments for torch.hub.load_state_dict_from_url() .

See also

Return type: None

abstract state_dict_url(framework)¶

Select URL of a downloadable state_dict.

Parameters: framework (str) – Name of the framework that was used to train the weights.
Raises: RuntimeError – If no state_dict is available.
Return type: str

VGG¶

class pystiche.enc.VGGMultiLayerEncoder(arch, **kwargs)¶

Multi-layer encoder based on VGG.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]

Parameters

arch (str) – VGG architecture. Has to match "vgg(11|13|16|19)(_bn)?".
pretrained – If True, loads builtin weights. Defaults to True.
framework – Name of the framework that was used to train the builtin weights. Defaults to "torch".
kwargs (Any) – Optional arguments of ModelMultiLayerEncoder .

Raises

RuntimeError – If pretrained is True and no weights are available for the combination of arch and framework.

pystiche.enc.vgg11_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 11.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG11 corresponds to configuration A in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

pystiche.enc.vgg11_bn_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 11 with batch normalization.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG11 corresponds to configuration A in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

pystiche.enc.vgg13_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 13.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG13 corresponds to configuration B in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

pystiche.enc.vgg13_bn_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 13 with batch normalization.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG13 corresponds to configuration B in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

pystiche.enc.vgg16_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 16.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG16 corresponds to configuration D in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

pystiche.enc.vgg16_bn_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 16 with batch normalization.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG16 corresponds to configuration D in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

pystiche.enc.vgg19_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 19.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG19 corresponds to configuration E in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

pystiche.enc.vgg19_bn_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on VGG 19 with batch normalization.

The VGG architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12]. VGG19 corresponds to configuration E in the paper.

Parameters: kwargs (Any) – Optional arguments of VGGMultiLayerEncoder .
Return type: VGGMultiLayerEncoder

AlexNet¶

class pystiche.enc.AlexNetMultiLayerEncoder(pretrained=True, framework='torch', internal_preprocessing=True, allow_inplace=False)¶

Multi-layer encoder based on AlexNet.

The AlexNet architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12].

Parameters

pretrained (bool) – If True, loads builtin weights. Defaults to True.
framework (str) – Name of the framework that was used to train the builtin weights. Defaults to "torch".
kwargs – Optional arguments of ModelMultiLayerEncoder .

Raises

RuntimeError – If pretrained and no weights are available for the
framework` –

pystiche.enc.alexnet_multi_layer_encoder(**kwargs)¶

Multi-layer encoder based on AlexNet.

The AlexNet architecture was introduced by Krizhevsky, Sutskever, and Hinton in [KSH12].

Parameters: kwargs (Any) – Optional arguments of AlexNetMultiLayerEncoder .
Return type: AlexNetMultiLayerEncoder

`pystiche.image`¶

Utilities¶

pystiche.image.calculate_aspect_ratio(image_size)¶

Return type: float

pystiche.image.image_to_edge_size(image_size, edge='short')¶

Return type: int

pystiche.image.edge_to_image_size(edge_size, aspect_ratio, edge='short')¶

Return type: Tuple[int, int]

pystiche.image.extract_batch_size(x)¶

Return type: int

pystiche.image.extract_num_channels(x)¶

Return type: int

pystiche.image.extract_image_size(x)¶

Return type: Tuple[int, int]

pystiche.image.extract_edge_size(x, edge='short')¶

Return type: int

pystiche.image.extract_aspect_ratio(x)¶

Return type: float

I/O¶

pystiche.image.read_image(file, device='cpu', make_batched=True, size=None, interpolation_mode='bilinear')¶

Read an image from file with PIL.Image and return it as Tensor .

Parameters

file (str) – Path to image file to be read.
device (Union[device, str]) – Device that the image is transferred to. Defaults to CPU.
make_batched (bool) – If True, a fake batch dimension is added to the image.
size (Optional[Union[int]]) – Optional size the image is resized to.
interpolation_mode (str) – Interpolation mode that is used to perform the optional resizing. Valid modes are "nearest", "bilinear", and "bicubic". Defaults to "bilinear".

Return type

pystiche.image.write_image(image, file, mode=None, **save_kwargs)¶

Write a Tensor image to a file with PIL.Image .

Parameters

image (Tensor) – Image to be written.
file (str) – Path to image file.
mode (Optional[str]) – Optional image mode. See the Pillow documentation for details.
**save_kwargs – Other parameters that are passed to PIL.Image.Image.save() .

Return type

pystiche.image.show_image(image, title=None, mode=None, size=None, interpolation_mode='bilinear')¶

Show an image and optionally read it from file first.

Note

show_image uses matplotlib.pyplot.imshow() as primary means to show images. If that is not available the native PIL.Image.Image.show() is used as a fallback.

Parameters

image (Union[Tensor, str]) – Image to be shown. If str this is treated as a path to an image and is read by read_image() .
title (Optional[str]) – Optional title of the image.
mode (Optional[str]) –
Optional image mode. See the Pillow documentation for details.
size (Optional[Union[int]]) – Optional size the image is resized to.
interpolation_mode (str) – Interpolation mode that is used to perform the optional resizing. Valid modes are "nearest", "bilinear", and "bicubic". Defaults to "bilinear".

Return type

Guides¶

pystiche.image.verify_guides(guides, verify_coverage=True, verify_overlap=True)¶

Verify if guides cover the whole canvas and if they do not overlap with each other.

Parameters

guides (Dict[str, Tensor]) – Guides to be verified.
verify_coverage (bool) – If True verifies that the guides cover the whole canvas. Defaults to True.
verify_overlap (bool) – If True verifies that the guides do not overlap with each other. Defaults to True.

Raises

RuntimeError – If at least on check is not successful.

Return type

pystiche.image.read_guides(dir, device='cpu', make_batched=True, size=None, interpolation_mode='nearest')¶

Read all guides from a directory using read_image() and return them as dictionary. The filename without extensions is used as region key.

Parameters

dir (str) – Path to root directory of the guide files.
device (Union[device, str]) – Device that the guides are transferred to. Defaults to CPU.
make_batched (bool) – If True, a fake batch dimension is added to every guide.
size (Optional[Union[int]]) – Optional size the guides are resized to.
interpolation_mode (str) – Interpolation mode that is used to perform the optional resizing. Valid modes are "nearest", "bilinear", and "bicubic". Defaults to "nearest".

Return type

Dict[str, Tensor]

pystiche.image.write_guides(guides, dir, ext='.png', mode='L', **save_kwargs)¶

Write guides to directory using write_image(). The region key is used as filename.

Parameters

guides (Dict[str, Tensor]) – Guides to be written.
dir (str) – Path to root directory of the guide files.
ext (str) – Extension that is appended to the filename. Defaults to ".png".
mode (str) –
Optional image mode. See the Pillow documentation for details. Defaults to "L".
**save_kwargs – Other parameters that are passed to PIL.Image.Image.save() .

Return type

pystiche.image.guides_to_segmentation(guides, color_map=None)¶

Combines multiple guides into one segmentation image.

Parameters

guides (Dict[str, Tensor]) – Guides to be combined.
color_map (Optional[Dict[str, Tuple[int, int, int]]]) – Optional mapping from regions to RGB triplets. If omitted, this color scheme is used for the region keys in alphabetical order.

Return type

pystiche.image.segmentation_to_guides(seg, region_map=None)¶

Splits a segmentation image in multiple guides.

Parameters

seg (Tensor) – Segmentation image to be split.
region_map (Optional[Dict[Tuple[int, int, int], str]]) – Optional mapping from RGB triplets to regions. If omitted, the RGB triplets are used as key in the output.

Return type

Dict[Union[Tuple[int, int, int], str], Tensor]

`pystiche.loss`¶

class pystiche.loss.Loss(*, encoder=None, input_guide=None, score_weight=1.0)¶

Bases: pystiche.core._modules.Module, abc.ABC

Abstract base class for all losses.

abstract forward(input_image)¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type: Union[Tensor, LossDict]

set_input_guide(guide)¶

Return type: None

class pystiche.loss.RegularizationLoss(*, encoder=None, input_guide=None, score_weight=1.0)¶

Bases: pystiche.loss._loss.Loss

Abstract base class for all regularization losses.

abstract calculate_score(input_repr)¶

Return type: Tensor

abstract input_enc_to_repr(enc)¶

Return type: Tensor

class pystiche.loss.ComparisonLoss(*, encoder=None, input_guide=None, target_image=None, target_guide=None, score_weight=1.0)¶

Bases: pystiche.loss._loss.Loss

Abstract base class for all comparison losses.

abstract calculate_score(input_repr, target_repr, *, ctx)¶

Return type: Tensor

abstract input_enc_to_repr(enc, ctx)¶

Return type: Tensor

set_target_image(image, *, guide=None)¶

Return type: None

abstract target_enc_to_repr(enc)¶

Return type: Tuple[Tensor, Optional[Tensor]]

Container¶

class pystiche.loss.LossContainer(named_losses, *, input_guide=None, target_image=None, target_guide=None, score_weight=1.0)¶

Generic container for Loss’es.

If called with an image, it will be passes it to all immediate losses and returns a pystiche.LossDict scaled with score_weight.

Parameters

named_losses (Sequence[Tuple[str, Loss]]) – Named immediate losses that will be called if OperatorContainer is called.
score_weight (float) – Score weight of the loss. Defaults to 1.0.

class pystiche.loss.MultiLayerEncodingLoss(mle, layers, encoding_loss_fn, *, layer_weights='mean', input_guide=None, target_image=None, target_guide=None, score_weight=1.0)¶

Convenience container for multiple Loss’es operating on different layers of the same pystiche.enc.MultiLayerEncoder.

Parameters

mle (MultiLayerEncoder) – Multi-layer encoder.
layers (Sequence[str]) – Layers of the mle that the children losses operate on.
encoding_loss_fn (Callable[[Encoder, float], Loss]) – Callable that returns a loss given a pystiche.enc.SingleLayerEncoder extracted from the mle and its corresponding layer weight.
layer_weights (Union[str, Sequence[float]]) – Weights passed to encoding_loss_fn. If "sum", each layer weight is set to 1.0. If "mean", each layer weight is set to 1.0 / len(layers). If sequence of float``s its length has to match ``layers’ length. Defaults to "mean".
score_weight (float) – Score weight of the loss. Defaults to 1.0.

Examples

>>> mle = pystiche.enc.vgg19_multi_layer_encoder()
>>> layers = ("relu1_1", "relu2_1", "relu3_1", "relu4_1", "relu5_1")
>>> loss = pystiche.loss.MultiLayerEncodingLoss(
...     mle,
...     layers,
...     lambda encoder, layer_weight: pystiche.loss.GramLoss(
...         encoder, score_weight=layer_weight
...     ),
... )
>>> input = torch.rand(2, 3, 256, 256)
>>> target = torch.rand(2, 3, 256, 256)
>>> loss.set_target_image(target)
>>> score = loss(input)

class pystiche.loss.MultiRegionLoss(regions, region_loss_fn, *, region_weights='sum', input_guide=None, target_image=None, target_guide=None, score_weight=1.0)¶

Convenience container for multiple Loss’es operating in different regions.

Parameters

regions (Sequence[str]) – Regions.
region_loss_fn (Callable[[str, float], Loss]) – Callable that returns a children loss given a region and its corresponding weight.
region_weights (Union[str, Sequence[float]]) – Weights passed to region_loss_fn. If "sum", each region weight is set to 1.0. If "mean", each region weight is set to 1.0 / len(layers). If sequence of float``s its length has to match ``regions’ length. Defaults to "mean".
score_weight (float) – Score weight of the loss. Defaults to 1.0.

Examples

>>> mle = pystiche.enc.vgg19_multi_layer_encoder()
>>> layers = ("relu1_1", "relu2_1", "relu3_1", "relu4_1", "relu5_1")
>>> def encoding_loss_fn(encoder, layer_weight):
...     return pystiche.loss.GramLoss(encoder, score_weight=layer_weight)
>>> regions = ("sky", "landscape")
>>> def region_loss_fn(region, region_weight):
...     return pystiche.loss.MultiLayerEncodingLoss(
...         mle,
...         layers,
...         encoding_loss_fn,
...         score_weight=region_weight,
...     )
>>> loss = pystiche.loss.MultiRegionLoss(regions, region_loss_fn)
>>> loss.set_regional_target_image("sky", torch.rand(2, 3, 256, 256))
>>> loss.set_regional_target_image("landscape", torch.rand(2, 3, 256, 256))
>>> input = torch.rand(2, 3, 256, 256)
>>> score = loss(input)

set_regional_input_guide(region, guide)¶

Invokes set_input_guide() on the operator of the given region.

Parameters

region (str) – Region.
guide (Tensor) – Input guide of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} 1 \times 1 \times H \times W$.

Return type

set_regional_target_image(region, image, guide=None)¶

Invokes set_target_image() on the operator of the given region.

Parameters

region (str) – Region.
image (Tensor) – Input guide of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times C \times H \times W$.
guide (Optional[Tensor]) –

Return type

class pystiche.loss.PerceptualLoss(content_loss, style_loss, regularization=None, *, content_image=None, content_guide=None, style_image=None, style_guide=None)¶

Perceptual loss comprising content and style loss as well as optionally a regularization.

Parameters

content_loss (Union[ComparisonLoss, LossContainer]) – Content loss.
style_loss (Union[ComparisonLoss, LossContainer]) – Guided style loss.
regularization (Optional[RegularizationLoss]) – Optional regularization.
content_image (Optional[Tensor]) – Content image applied as target image to the content_loss.
content_guide (Optional[Tensor]) – Content guide applied as input guide to the style_loss.
style_image (Optional[Tensor]) – Style image applied as target image to style_loss.
style_guide (Optional[Tensor]) – Style guide applied as target image to style_loss.

regional_content_guide(region=None)¶

Regional content guide.

Parameters: region (Optional[str]) – Region to get the content guide from.
Return type: Optional[Tensor]

regional_style_image(region=None)¶

Regional style image.

Parameters: region (Optional[str]) – Region to get the style image from.
Return type: Optional[Tensor]

set_content_guide(guide, *, region=None)¶

Sets the content guide.

Parameters

guide (Tensor) – Content guide.
region (Optional[str]) – Optional region to set the guide for. If omitted, the guide will be applied to all regions.

Return type

set_style_image(image, *, guide=None, region=None)¶

Sets the style image and guide.

Parameters

image (Tensor) – Style image.
guide (Optional[Tensor]) – Style guide.
region (Optional[str]) – Optional region to set the image and guide for. If omitted, the image and guide will be applied to all regions.

Return type

Regularization¶

class pystiche.loss.TotalVariationLoss(*, exponent=2.0, input_guide=None, score_weight=1.0)¶

Bases: pystiche.loss._loss.RegularizationLoss

The total variation loss is a regularizer used to suppress checkerboard artifacts by penalizing the gradient of the image. It is calculated by

\[\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \mean[\limits_{i,j}] \parentheses{\parentheses{x_{i,j+1} - x_{i,j}}^2 + \parentheses{x_{i+1,j} - x_{i,j}}^2}^{\frac{\beta}{2}}\]

where $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} x$ denotes the image and $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} i,j$ index a specific pixel.

Note

Opposed to the paper, the implementation calculates the grand average $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \mean$ opposed to the grand sum $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \sum$ to account for different sized images.

Parameters

exponent (float) – Parameter $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \beta$ . A higher value leads to more smoothed results. Defaults to 2.0.
score_weight (float) – Score weight of the operator. Defaults to 1.0.

Examples

>>> loss = pystiche.loss.TotalVariationLoss()
>>> input = torch.rand(2, 3, 256, 256)
>>> score = loss(input)

See also

The total variation loss was introduced by Mahendran and Vedaldi in [MV15] .

Comparison¶

class pystiche.loss.FeatureReconstructionLoss(encoder, *, input_guide=None, target_image=None, target_guide=None, score_weight=1.0)¶

Bases: pystiche.loss._loss.ComparisonLoss

The feature reconstruction loss is the de facto standard content loss. It measures the mean squared error (MSE) between the encodings of an input_image $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \hat{I}$ and a target_image $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} I$ :

\[\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \mean \parentheses{\Phi\of{\hat{I}} - \Phi\of{I}}^2\]

Note

Parameters

encoder (Encoder) – Encoder $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \Phi$.
score_weight (float) – Score weight of the operator. Defaults to 1.0.

Examples

>>> mle = pystiche.enc.vgg19_multi_layer_encoder()
>>> encoder = mle.extract_encoder("relu4_2")
>>> loss = pystiche.loss.FeatureReconstructionLoss(encoder)
>>> input = torch.rand(2, 3, 256, 256)
>>> target = torch.rand(2, 3, 256, 256)
>>> loss.set_target_image(target)
>>> score = loss(input)

See also

The feature reconstruction loss was introduced by Mahendran and Vedaldi in [MV15] , but its name was coined by Johnson, Alahi, and Fei-Fei in [JAL16] .

class pystiche.loss.GramLoss(encoder, *, normalize=True, input_guide=None, target_image=None, target_guide=None, score_weight=1.0)¶

Bases: pystiche.loss._loss.ComparisonLoss

The gram loss is a style loss based on the correlation of feature map channels. It measures the mean squared error (MSE) between the channel-wise Gram matrices of the encodings of an input_image $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \hat{I}$ and a target_image $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} I$:

\[\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \mean \parentheses{\fun{gram}{\Phi\of{\hat{I}}} - \fun{gram}{\Phi\of{I}}}^2\]

Here $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \Phi\of{\cdot}$ denotes the encoder and $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \fun{gram}{\cdot}$ denotes pystiche.gram_matrix().

Note

Parameters

encoder (Encoder) – Encoder $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \Phi\left( \cdot \right)$.
normalize (bool) – If True, normalizes the Gram matrices to account for different sized images. See pystiche.gram_matrix() for details. Defaults to True.
score_weight (float) – Score weight of the operator. Defaults to 1.0.

Examples

>>> mle = pystiche.enc.vgg19_multi_layer_encoder()
>>> encoder = mle.extract_encoder("relu4_2")
>>> loss = pystiche.loss.GramLoss(encoder)
>>> input = torch.rand(2, 3, 256, 256)
>>> target = torch.rand(2, 3, 256, 256)
>>> loss.set_target_image(target)
>>> score = loss(input)

See also

The feature reconstruction loss was introduced by Gatys, Ecker, and Bethge in [GEB16] .

class pystiche.loss.MRFLoss(encoder, patch_size, *, stride=1, target_transforms=None, input_guide=None, target_image=None, target_guide=None, score_weight=1.0)¶

Bases: pystiche.loss._loss.ComparisonLoss

The MRF loss is a style loss based on Markov Random Fields (MRFs). It measures the mean squared error (MSE) between neural patches extracted from the encodings of an input_image $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \hat{I}$ and a target_image $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} I$:

\[\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \mean \parentheses{p_n\of{\Phi\of{\hat{I}}} - p_{MCS\of{n}}\of{\Phi\of{\hat{I}}}}^2\]

Since the number of patches might be different for both images and the order of the patches does not correlate with the order of the enclosed style element, for each input neural patch $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} n$ a fitting target neural patch is to selected based on the maximum cosine similarity $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} MCS\of{n}$ with pystiche.cosine_similarity().

Note

Parameters

encoder (Encoder) – Encoder $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \Phi$ .
patch_size (Union[int, Sequence[int]]) – Spatial size of the neural patches.
stride (Union[int, Sequence[int]]) – Distance between two neural patches.
target_transforms (Optional[Iterable[Module]]) – Optional transformations to apply to the target image before the neural patches are extracted. Defaults to None.
score_weight (float) – Score weight of the operator. Defaults to 1.0.

Examples

>>> mle = pystiche.enc.vgg19_multi_layer_encoder()
>>> encoder = mle.extract_encoder("relu4_2")
>>> patch_size = 3
>>> loss = pystiche.loss.MRFLoss(encoder, patch_size)
>>> input = torch.rand(2, 3, 256, 256)
>>> target = torch.rand(2, 3, 256, 256)
>>> loss.set_target_image(target)
>>> score = loss(input)

See also

The MRF loss was introduced by Li and Wand in [LW16].

static scale_and_rotate_transforms(num_scale_steps=1, scale_step_width=0.05, num_rotate_steps=1, rotate_step_width=10.0)¶

Generate a list of scaling and rotations transformations.

See also

The output of this method can be used as parameter target_transforms of MRFLoss to enrich the space of target neural patches:

target_transforms = MRFOperator.scale_and_rotate_transforms()
loss = pystiche.loss.MRFLoss(..., target_transforms=target_transforms)

Parameters

num_scale_steps (int) – Number of scale steps. Each scale is performed in both directions, i.e. enlarging and shrinking the motif. Defaults to 1.
scale_step_width (float) – Width of each scale step. Defaults to 5e-2.
num_rotate_steps (int) – Number of rotate steps. Each rotate is performed in both directions, i.e. clockwise and counterclockwise. Defaults to 1.
rotate_step_width (float) – Width of each rotation step in degrees. Defaults to 10.0.

Return type

List[ScaleAndRotate]

Returns

(num_scale_steps * 2 + 1) * (num_rotate_steps * 2 + 1) transformations in total comprising every combination given by the input parameters.

`pystiche.loss.functional`¶

pystiche.loss.functional.mrf_loss(input, target, eps=1e-08, reduction='mean', batched_input=None)¶

Calculates the MRF loss. See pystiche.loss.MRFLoss for details.

Parameters

input (Tensor) – Input of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times S_1 \times N_1 \times \dots \times N_D$.
target (Tensor) – Target of shape $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} B \times S_2 \times N_1 \times \dots \times N_D$.
eps (float) – Small value to avoid zero division. Defaults to 1e-8.
reduction (str) – Reduction method of the output passed to pystiche.misc.reduce(). Defaults to "mean".
batched_input (Optional[bool]) – If False, treat the first dimension of the inputs as sample dimension, i.e. $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} S \times N_1 \times \dots \times N_D$. Defaults to True. See pystiche.cosine_similarity() for details.

Examples

>>> import pystiche.loss.functional as F
>>> input = torch.rand(1, 256, 64, 3, 3)
>>> target = torch.rand(1, 128, 64, 3, 3)
>>> score = F.mrf_loss(input, target, batched_input=True)

Return type: Tensor

pystiche.loss.functional.total_variation_loss(input, exponent=2.0, reduction='mean')¶

Calculates the total variation loss. See pystiche.ops.TotalVariationOperator for details.

Parameters

input (Tensor) – Input image
exponent (float) – Parameter $\newcommand{\parentheses}[1]{\left( #1 \right)} \newcommand{\brackets}[1]{\left[ #1 \right]} \newcommand{\mean}[1][]{\overline{\sum #1}} \newcommand{\fun}[2]{\text{#1}\of{#2}} \newcommand{\of}[1]{\parentheses{#1}} \newcommand{\dotproduct}[2]{\left\langle #1 , #2 \right\rangle} \newcommand{\openinterval}[2]{\parentheses{#1, #2}} \newcommand{\closedinterval}[2]{\brackets{#1, #2}} \beta$ . A higher value leads to more smoothed results. Defaults to 2.0.
reduction (str) – Reduction method of the output passed to pystiche.misc.reduce(). Defaults to "mean".

Examples

>>> import pystiche.loss.functional as F
>>> input = torch.rand(2, 3, 256, 256)
>>> score = F.total_variation_loss(input)

Return type: Tensor

pystiche.loss.functional.value_range_loss(input, min=0.0, max=1.0, reduction='mean')¶

Return type: Tensor

`pystiche.meta`¶

pystiche.meta.tensor_meta(x, **kwargs)¶

Return type: Dict[str, Any]

pystiche.meta.is_scalar_tensor(x)¶

Return type: bool

pystiche.meta.is_conv_module(x)¶

Return type: bool

pystiche.meta.conv_module_meta(x, **kwargs)¶

Return type: Dict[str, Any]

pystiche.meta.is_pool_module(x)¶

Return type: bool

pystiche.meta.pool_module_meta(x, **kwargs)¶

Return type: Dict[str, Any]

`pystiche.misc`¶

pystiche.misc.get_input_image(starting_point='content', content_image=None, style_image=None)¶

Generates an input image for NST from the given starting_point.

Parameters

starting_point (Union[str, Tensor]) – If Tensor returns a copy. If "content" or "style" returns a copy of content_image or style_image, respectively. If "random" returns a white noise image with the dimensions of content_image or style_image, respectively. Defaults to "content".
content_image (Optional[Tensor]) – Content image. Only required if starting_point is "content" or "random".
style_image (Optional[Tensor]) – Style image. Only required if starting_point is "style" or "random".

Return type

pystiche.misc.get_device(device=None)¶

Selects a device to perform an NST on.

Parameters: device (Optional[str]) – If str, returns the corresponding device. If None selects CUDA if available and otherwise CPU. Defaults to None.
Return type: device

pystiche.misc.reduce(x, reduction)¶

Reduces a Tensor as specified.

Parameters

x (Tensor) – Input tensor.
reduction (str) – Reduction method to be applied to x. If "none", no reduction will be applied. If "sum" or "mean", the sum() or mean() will be applied across all dimensions of x.

Return type

pystiche.misc.build_complex_obj_repr(name, properties=None, named_children=(), line_length=80, num_indent=2)¶

Return type: str

`pystiche.optim`¶

pystiche.optim.default_image_optimizer(input_image)¶

Parameters: input_image (Tensor) – Image to be optimized.
Return type: LBFGS
Returns: torch.optim.LBFGS optimizer with a learning rate of 1.0. The pixels of input_image are set as optimization parameters.

pystiche.optim.image_optimization(input_image, criterion, optimizer=None, num_steps=500, preprocessor=None, postprocessor=None, quiet=False)¶

Perform an image optimization with integrated logging.

Parameters

input_image (Tensor) – Image to be optimized.
criterion (Module) – Optimization criterion.
optimizer (Union[Optimizer, Callable[[Tensor], Optimizer], None]) – Optional optimizer or optimizer getter. If omitted, default_image_optimizer() is used. If a preprocessor is used, has to be a getter.
num_steps (Union[int, Iterable[int]]) – Number of optimization steps. Defaults to 500.
preprocessor (Optional[Module]) – Optional preprocessor that is called with the input_image before the optimization.
postprocessor (Optional[Module]) – Optional preprocessor that is called with the input_image after the optimization.
quiet (bool) – If True, no information is printed to STDOUT during the optimization. Defaults to False.

Raises

RuntimeError – If preprocessor is used and optimizer is not passed as
getter. –

Return type

pystiche.optim.pyramid_image_optimization(input_image, criterion, pyramid, get_optimizer=None, preprocessor=None, postprocessor=None, quiet=False)¶

Perform a image optimization for pystiche.pyramid.ImagePyramid s with integrated logging.

Parameters

input_image (Tensor) – Image to be optimized.
criterion (Module) – Optimization criterion.
pyramid (ImagePyramid) – Image pyramid.
get_optimizer (Optional[Callable[[Tensor], Optimizer]]) – Optional getter for the optimizer. If None, default_image_optimizer() is used. Defaults to None.
preprocessor (Optional[Module]) – Optional preprocessor that is called with the input_image before the optimization.
postprocessor (Optional[Module]) – Optional preprocessor that is called with the input_image after the optimization.
quiet (bool) – If True, no information is printed to STDOUT during the optimization. Defaults to False.

Return type

pystiche.optim.default_model_optimizer(transformer)¶

Parameters: transformer (Module) – Transformer to be optimized.
Return type: Optimizer
Returns: torch.optim.Adam optimizer with a learning rate of 1e-3. The parameters of transformer are set as optimization parameters.

pystiche.optim.model_optimization(image_loader, transformer, criterion, criterion_update_fn=None, optimizer=None, quiet=False)¶

Perform a model optimization for a single epoch with integrated logging.

Parameters

image_loader (DataLoader) – Images used as input for the transformer. Drawing from this should yield either an batched image or a tuple or list with a batched image as first item.
transformer (Module) – Transformer to be optimized.
criterion (Module) – Optimization criterion.
criterion_update_fn (Optional[Callable[[Tensor, Module], None]]) – Is called before each optimization step with the current images and the optimization criterion. If omitted and criterion is a PerceptualLoss or a GuidedPerceptualLoss this defaults to invoking set_content_image().
optimizer (Optional[Optimizer]) – Optional optimizer. If None, default_model_optimizer() is used.
quiet (bool) – If True, no information is printed to STDOUT during the optimization. Defaults to False.

Return type

Module

pystiche.optim.multi_epoch_model_optimization(image_loader, transformer, criterion, criterion_update_fn=None, epochs=2, optimizer=None, lr_scheduler=None, quiet=False)¶

Perform a model optimization for multiple epochs with integrated logging.

Parameters

image_loader (DataLoader) – Images used as input for the transformer. Drawing from this should yield either an batched image or a tuple or list with a batched image as first item.
transformer (Module) – Transformer to be optimized.
criterion (Module) – Optimization criterion.
criterion_update_fn (Optional[Callable[[Tensor, Module], None]]) – Is called before each optimization step with the current images and the optimization criterion. If omitted and criterion is a PerceptualLoss or a GuidedPerceptualLoss this defaults to invoking set_content_image().
epochs (int) – Number of epochs. Defaults to 2.
optimizer (Optional[Optimizer]) – Optional optimizer. If None, it is extracted from lr_scheduler or func:default_model_optimizer is used.
lr_scheduler (Optional[_LRScheduler]) – Optional learning rate scheduler. step() is invoked after every epoch.
quiet (bool) – If True, no information is printed to STDOUT during the optimization. Defaults to False.

Return type

Module

`pystiche.pyramid`¶

Image pyramid¶

class pystiche.pyramid.ImagePyramid(edge_sizes, num_steps, edge='short', interpolation_mode='bilinear', resize_targets=())¶

Image pyramid for a coarse-to-fine optimization on different levels. If iterated on yields PyramidLevel s and handles the resizing of all set images and guides of resize_targets.

Parameters

edge_sizes (Sequence[int]) – Edge sizes for each level.
num_steps (Union[Sequence[int], int]) – Number of steps for each level. If sequence of int its length has to match the length of edge_sizes.
edge (Union[Sequence[str], str]) – Corresponding edge to the edge size for each level. Can be "short" or "long". If sequence of str its length has to match the length of edge_sizes. Defaults to "short".
interpolation_mode (str) –
Interpolation mode used for the resizing of the images. Defaults to "bilinear".

Note

For the resizing of guides "nearest" is used regardless of the interpolation_mode.
resize_targets (Collection[Loss]) – Targets for resizing of set images and guides during iteration.

class pystiche.pyramid.OctaveImagePyramid(max_edge_size, num_steps, num_levels=None, min_edge_size=64, **image_pyramid_kwargs)¶

Bases: pystiche.pyramid.pyramid.ImagePyramid

Image pyramid that comprises levels spaced by a factor of two.

Parameters

max_edge_size (int) – Maximum edge size.
num_steps (Union[int, Sequence[int]]) –
Number of steps for each level.

Note

If num_steps is specified as sequence of int``s, you should also specify ``num_levels to match the lengths
num_levels (Optional[int]) – Optional number of levels. If None, the number is determined by the number of steps of factor two between max_edge_size and min_edge_size.
min_edge_size (int) – Minimum edge size for the automatic calculation of num_levels.
image_pyramid_kwargs (Any) – Additional options. See ImagePyramid for details.

Pyramid level¶

class pystiche.pyramid.PyramidLevel(edge_size, num_steps, edge)¶

Level with an pystiche.pyramid.ImagePyramid. If iterated on, yields the step beginning at 1 and ending in num_steps.

Parameters

edge_size (int) – Edge size.
num_steps (int) – Number of steps.
edge (str) – Corresponding edge to the edge size. Can be "short" or "long".

resize_guide(guide, aspect_ratio=None, interpolation_mode='nearest')¶

Resize a guide to the edge_size on the corresponding edge of the PyramidLevel.

Parameters

guide (Tensor) – Guide to be resized.
aspect_ratio (Optional[float]) – Optional aspect ratio of the output. If None, the aspect ratio of guide is used. Defaults to None.
interpolation_mode (str) – Interpolation mode used to resize image. Defaults to "nearest".

Return type