CNN Pretrained Model with Shape Bias using Image Decomposition

Convolutional Neural Networks (CNNs) were originally designed to imitate the human visual system [10,22,23] and are utilized for numerous tasks, such as classification, semantic segmentation, and image generation. Although CNNs mimic the human visual system, there are still differences [6,9,13,15,32]. One notable difference between humans and CNNs is that humans tend to recognize objects by their global shape, whereas CNNs prioritize local texture over global shape.

The human visual cortex tends to first capture edges and points, then the shapes of objects, and finally the entire image containing the objects [24]. When humans recognize an object, they tend to focus on the global shape information of the entire object rather than on the local texture information of a part of the object. For example, when humans recognize elephants, they rarely focus on the shape of the wrinkles on their body surface but tend to recognize the body shape and positions of body parts, such as the nose and ears.

In contrast, Convolutional Neural Networks (CNNs) recognize objects primarily based on their texture, thanks to the use of convolutional layers that calculate correlations between the central pixel and its neighboring pixels to extract local texture features, rather than global shape. The pooling layer further expands the recognition range by down-sampling the feature maps. The CNN architecture gradually expands the recognition area from small to large by stacking convolutional and pooling layers. However, the local texture features obtained in the early layers tend to dominate the overall object recognition performance, as they influence the operations of the later layers. Additionally, as the kernel size per layer is usually small, the spatial receptive field is limited, and even if many layers are stacked on top of each other, the receptive field of CNNs does not become very large [5]. In other words, humans tend to focus on global shape information for object recognition, while CNNs tend to focus on local texture information for object recognition. This tendency to focus on global shapes rather than local textures in the analysis is known as shape bias, and the opposite is called texture bias.

Developing CNN with a shape bias has many advantages. One is that the CNN can grasp features more accurately for shape-dominant image data, such as cartoons or sketch images. In addition, focusing on shape also makes object judgments less susceptible to noise, blurring, and other image degradation, which may counter hostile attacks that may add perturbation and cause misclassification. Such a bias can also improve the efficiency and accuracy of CNN transfer learning and fine-tuning by selecting a model with an appropriate bias for transferring learning data. Furthermore, by analyzing the area in the image wherein the CNNs identify the shape and texture, we can improve their explainability.

In a previous study, Geirhos et al. [12] created Stylized-ImageNet, which is an image dataset that has different labels for local textures and global shapes, and experimentally confirmed that CNNs have a texture bias. They also used the Stylized-ImageNet to create a model with shape bias. Hermann et al. [20] performed various data augmentation techniques to quantitatively evaluate texture and shape biases. However, these previous studies not only changed the texture but also the shape of the object, or generated unrealistic images, which may negatively impact learning and raise questions about whether the shape is correctly recognized.

This study used image decomposition to remove texture from an image and used this decomposition to create an image dataset with information that is more meaningful for shape than texture. Thereafter, we trained the CNN on the dataset such that it had more bias toward the shape. Through experiments, we confirmed that the model had a shape bias, and fine-tuning the shape-dominant dataset resulted in the best performance compared to pretraining the model on other datasets.

The main contributions of this research can be summarized in the following two points:

• This study shows that the shape bias of a Neural Network can be increased with a shape-dominant image dataset obtained by image decomposition. The conventional dataset employed by Geirhos et al. [12] uses specialized techniques for dataset creation, and it is considered very time-consuming or even hard to apply to a different modality. By contrast, our method utilizes simple image decomposition techniques, providing the significant advantage of being able to create shape-biased data from any dataset and train new models with enhanced shape biases.

• While Geirhos et al. [12] aimed to create shape-biased models, the conventional approach did not demonstrate significant improvements in recognition performance on images from domains outside of ImageNet. In contrast, our method demonstrates enhanced recognition performance on datasets with limited texture information, which represents a significant step forward in the field.

The rest of this paper is organized as follows: Section 2 introduces related work on CNN bias and datasets. Section 3 describes the proposed method and pretraining settings. In Section 4, we explain the four experiments of the proposed method and discuss future work. Finally, Section 5 concludes the paper.

ImageNet [28] is a well-known dataset used for pertaining and includes ImageNet-lk [28] and ImageNet-21k [4], comprising 1 and 14 million images, respectively. It also has a wide variety of evaluation sets. ImageNet-C [18], where the images have been processed in various ways, such as adding noise, to verify model robustness. ImageNet-A or ImageNet-O [19], which is a collection of images originally included in ImageNet, is prone to misclassifications. ImageNet-R [17] contains a collection of images included in ImageNet, which are artifacts rather than real objects, such as pictures or dolls. ImageNetSketch [35]is composed of sketch images of ImageNet labels. It also contains the larger and publicly unavailable JFT-300M [31], which consists of 300 million images collected independently by Google; it is used for pretraining the recently introduced ViT [7].

Studies have shown that CNN models trained using ImageNet [28] prefer local texture information to global shape information. Baker et al. [2] divided images into global shapes (edges and silhouettes) and local textures (parts of edges and within edges), and investigated the changes in recognition accuracy between humans and CNNs by varying each of these. Their results showed that the recognition accuracy of humans deteriorated significantly when the global shape was corrupted compared to when the local texture was corrupted. In the case of CNNs, the recognition accuracy did not deteriorate even if the global shape was corrupted, whereas it deteriorated significantly when the local texture was corrupted. Geirhos et al. [12] applied a style transformation to ImageNet to create image datasets with different shapes and textures and trained the CNN such that it had a shape bias. In the experiment, they measured the bias by performing image classification using two correct labels, shape and texture, for both humans and CNNs. The results showed that the human classifier focused on the global shape, whereas the original CNN focused on local texture. In contrast, models trained to have a shape bias had a bias similar to that of humans. Geirhos et al. [11] also compared various architectures and learning methods for humans, CNNs, and ViTs. They showed that learning on large datasets over long periods of time and learning with noise labels and data increases the bias and improves noise tolerance.

Tuli et al. [34] evaluated more accurate models by investigating the error consistency of deep neural networks. They found that ViTs have a more human-like bias than CNNs and a similar error consistency. Hermann et al. [20] investigated the impact of data augmentation and learning goals on shape bias and found that CNNs can lose shape information in their architecture as they learn. Shi et al. [29] proposed a vision-inspired algorithm to reduce the texture bias of CNNs and found that other algorithms, such as adversarial learning, can be used to improve recognition accuracy. Azad et al. [1] achieved the best results on several datasets for few-shot segmentation by reducing texture bias. Ding et al. [5] showed that increasing the kernel size of a CNN increases its spatial receptive field and shape bias. Raghu et al. [27] mapped the similarity between the layers of a CNN and ViT and showed that the CNN acquires global features from local features, whereas the ViT acquires global features from the beginning.

Many of these related studies, which attempted to obtain shape bias by modifying datasets, largely corrupted the shape of the images in the dataset. Therefore, CNNs may not have acquired the shape correctly. In this study, the image was smoothed to remove the texture while preserving the shape such that the CNN could acquire the shape from the image.

The proposed method involves decomposing images into shape and texture components, creating and training a shape image dataset, and performing fine-tuning on datasets with shape dominance. An overview of the proposed method is shown in Figure 1. In this study, we initially decompose the pretraining images into local texture and global shape images by conventional image decomposition techniques and then reconstruct the pretraining dataset using only the global shape images. Subsequently, we pretrain the neural network using only the training images to obtain the shape bias. Finally, we quantitatively evaluate the bias through four types of experiments and demonstrate the superiority of our method for shape-dominant image data.

In the previous study by Geirhos et al. [12], images are generated using style transfer [21]. Style transfer is used to change the local texture of an image to a different local texture that is not meaningful for the correct label, thereby increasing the importance of global shape information. However, style transfer often produces unrealistic images because changing the local texture also affects the global shape, which negatively impacts learning and reduces accuracy. This is similar to learning from noisy images and has been a major concern.

In this study, we propose a method for constructing a shape-dominant dataset by using classical image decomposition methods to separate the shapes and textures of each image. We use two different methods to decompose the images: the edge-preserving smoothing algorithm proposed by Subr et al. [30]and L0-Smoothing proposed by Xu et al. [38]. First, the edge-preserving smoothing algorithm defines the oscillation between the local minima and maxima in an image in detail (local texture). Then, the local minima and maxima in the image are calculated, and the envelope connecting each extreme value is obtained. Finally, we estimate the average of the envelopes of the local minima and maxima, and generate a smoothed image S. Next, L0-Smoothing reduces the number of pixels with non-zero differences between adjacent pixels, resulting in a smoothed image.

Using an algorithm that preserves the edges of images and maintains their shape, images can be added together. Let the original image be I, global shape image be S, and local texture image be T. We decompose I into S and T such that I = S + T, and reconstruct the image using the weighted combination as follows:

Thus, the shape-dominant image I’ is obtained. An example of images I’ generated by applying the edge-preserving smoothing algorithm is shown in Figure 2. This process (α = 0) was applied to all ImageNet images to create datasets Smooth-ImageNet (edge-preserving smoothing algorithm) and L0-ImageNet (L0-Smoothing).

Figure 3 illustrates some examples from the original ImageNet and Stylized-ImageNet datasets [12], as well as sample images from the Smooth-ImageNet and L0-ImageNet datasets. For the convenience of using style transfer [21], the images in Stylized-ImageNet were resized to (256, 256). The styles used for style transfer were based on paintings [26] primarily collected from a website called WikiArt.¹ However, due to the gaps between domains and the accuracy of the style transfer itself, the images in the Stylized-ImageNet dataset may not be interpretable to the human eye. In such cases, the image may contain a mismatch between the shape and texture, which could lead to inaccuracies in CNN’s ability to acquire the shape of the original image. In contrast, the proposed datasets employ an algorithm that preserves the edges of the image while removing the texture. By training the CNN without texture information, the shape of the image is preserved, allowing humans to immediately determine the contents of the image. Comparing the Smooth-ImageNet and LO-ImageNet datasets, LO-ImageNet is smoother than Smooth-ImageNet. In terms of processing speed, creating Smooth-ImageNet required more than one month, whereas L0-ImageNet was created in approximately three days.

We created a pretrained model of ResNet50 [16] using the proposed datasets (Smooth-ImageNet and L0-ImageNet), ImageNet as a comparison method, and Stylized-ImageNet, for a total of four types of data. For the pretraining dataset, we used only ImageNet,1,281,167 training images and 50,000 validation images from ImageNet. We trained for 90 epochs with a batch size of 256 and an initial learning rate of 0.1. The learning rate was divided by 10 for 30,60, and 80 epochs. For the other three datasets, both the dataset and ImageNet were used for training. For example, when training Smooth-ImageNet, we used images from both ImageNet and Smooth-ImageNet, and 2,562,334 training images. Stochastic gradient descent (SGD) and cross-entropy loss were used.

In this Section, we quantitatively evaluate the shape bias proposed in previous studies. We also evaluate the efficiency of the proposed model with the shape bias by conducting experiments on three datasets characterized by shapes that are not related to either the proposed or comparative method.

In the first experiment, we quantitatively compared the shape bias of models trained on the proposed datasets, Smooth-ImageNet and L0-ImageNet, with those trained on the conventional datasets, ImageNet and Stylized-ImageNet. In this experiment, we tested the degree to which increasing the shape bias degrades the accuracy of conventional ImageNet classification with respect to ImageNet. Four models (Smooth-ImageNet, L0-ImageNet, original ImageNet, and Stylized-ImageNet), and three models that were fine-tuned with ImageNet for each trained model, were compared and evaluated. Pretraining was performed using the same parameter settings as those presented in Section 3.2, and the original ImageNet was used for fine-tuning. We fine-tuned for 10 epochs with a learning rate of 0.01.

We also compared and evaluated the shape bias and accuracy of the latest image recognition models, ConvNeXt-T [25] and EfficientNetV2-S [33]. The pre-training for ConvNeXt-T was performed using the official pre-training settings, and for EfficientNetV2-S, the settings in [37] were used as a reference. We fine-tuned for 30 epochs.

To quantitatively evaluate bias, we used the criteria proposed by Geirhos et al. [12]. As test data, images were style-transformed using a method similar to that used for Stylized-ImageNet to generate images with two different labels for global shapes and local textures (e.g., car for global shapes and cat for local textures). They used 16 classes consisting of airplanes, bears, bicycles, birds, boats, bottles, cars, cats, chairs, clocks, dogs, elephants, keyboards, knives, ovens, and trucks. They defined the number of shape matches (NoSM) as the number of model outputs that match the labels for the global shapes and the number of texture matches (NoTM) as the number of model outputs that match the labels for the local textures. Using these two quantities, they calculated the degree of shape bias as follows:

This metric evaluates the shape bias for each model and the accuracy of the original ImageNet validation set.

The shape biases and recognition rates are shown in Table 1. For shape bias, Stylized-ImageNet had the highest value, followed by L0-ImageNet, Smooth-ImageNet, and ImageNet. Note that because the test data evaluated for the shape bias were created using the same method that used for creating Stylized-ImageNet, this experiment can be considered to have been conducted under favorable conditions for Stylized-ImageNet. In addition, the models trained on the proposed Smooth-ImageNet and L0-ImageNet datasets had larger shape bias values than those trained on the original ImageNet. Furthermore, when ImageNet was fine-tuned, the model trained on Stylized-ImageNet showed a slight decrease in shape bias, whereas those of models trained on Smooth-ImageNet and L0-ImageNet did not decrease.

Tables 2 and 3 shows the shape bias and accuracy of the latest image recognition models, ConvNeXt-T and EfficientNetV2-S. In both cases, the shape bias increased when trained on the proposed dataset and did not decrease significantly after fine-tuning. The accuracy is higher for the training with ImageNet alone, but the difference is not significant after fine-tuning. The proposed method has higher accuracy than Stylized-Imagenet, although there is a decrease in shape bias.

When the proposed datasets (Smooth-ImageNet and L0-ImageNet) were used for pre-training and further fine-tuning using ImageNet, TOP-1 and TOP-5 showed almost equal or better accuracy compared to training using ImageNet. In addition, the shape bias in our method did not decrease significantly after fine-tuning compared to Stylized-ImageNet. Thus, compared to ImageNet, the proposed datasets can acquire shape bias and successfully improve or maintain accuracy without decreasing shape bias.

In the second experiment, we performed anime image recognition as an example to show that shape bias can improve classification performance. We used anime images to evaluate the effectiveness of the proposed method. Because anime images have fewer local textures than natural images, they can be considered shape-dominant. Therefore, it can be used to determine the effectiveness of a pretrained model with a shape bias for shape-dominant data.

We used the AnimeFace Character dataset;² 11,592 out of 14,490 images were used as training data, and 2,898 images were used as test data.

Samples of images in this dataset are shown in Figure 4. The number of classes is 203, and each class contains approximately 50–100 images. For fine-tuning, we used the SGD optimizer and cross-entropy loss as the loss function. The batch size was 64 and the learning rate was 0.01. The model was trained for 40 epochs.

The experimental results are shown in Table 4. The models pretrained on the proposed L0-ImageNet had the highest TOP-1 and TOP-5, followed by Smooth-ImageNet and Stylized-ImageNet; the models pretrained on the original ImageNet achieved the lowest accuracy. The model pretrained with L0-ImageNet was the most accurate for animated images with few textures. For the validation set of ImageNet, the model trained on the original ImageNet was the most accurate at the pretraining stage. However, in the fine-tuning of animated images, the accuracy of the proposed method is considered superior because of its emphasis on shape. This indicates that the proposed method is superior for animated images, which comprise shape-dominated data. Conversely, the accuracy of Stylized-ImageNet may be lower owing to the presence of different local textures resulting from its data-creation method.

In the third experiment, by performing a sketch image recognition task, we showed that shape bias can improve classification performance. Sketches mainly consist of global shapes of an object and have little meaningful local texture. They have less local texture information than anime images and no color information. Therefore, the data is shape-dominant compared to natural images. We confirmed the effectiveness of the proposed method on these shape-dominant data, which are different from those of anime images.

For fine-tuning, we used a dataset comprising a collection of sketched images created by Eitz et al. [8]. Some samples from the dataset are shown in Figure 5. Of the 20,000 images,16,000 were used for training and 4,000 were used for testing. The dataset comprises 250 classes, and the learning rate for fine-tuning is 0.01. SGD was used as the optimizer and cross-entropy loss was used as the loss function. The batch size was 128.

The experimental results are shown in Table 5. Similar to the case of animated images, the models trained on L0-ImageNet and the proposed method showed the highest accuracy, followed by those trained on Smooth-ImageNet, Stylized-ImageNet, and original ImageNet. The group of models with low accuracy in the ImageNet validation set became more accurate for the sketched images. Thus, we created an effective pretrained model for data without color information. Furthermore, the models trained using the proposed method were found to be more effective for sketch images, which have even less local texture information than animated images.

In this experiment, we used logo images as examples of shape-dominant images. A logo is an illustration of a text string or object, and logos are applied to various items, such as organizations and products. They have a few high-frequency components similar to animated images, and some comprise different colors for different backgrounds or objects. Therefore, they can be considered color-independent shape images.

The Logo-2k+ dataset [36] was used in this experiment. Samples of images in the dataset are shown in Figure 6. The Logo-2k+ dataset consists of 10 parent categories and 2,341 child categories for a total of 167,140 images. Of the 167,140 images,116,958 were used as training data and 50,182 as test data. The learning rate was set to 0.01, a cosine scheduler was used, and the warm-up time was set to five epochs. SGD was used as the optimizer and cross-entropy loss was used as the loss function. The batch size was 128.

The experimental results are shown in Table 6. Similar to the aforementioned two datasets, the accuracy of the model trained on the proposed dataset was the highest, followed by Stylized-ImageNet and the original ImageNet. Thus, the proposed model was also effective for datasets containing logo images, where shape information is more important than color information.

These results indicate that compared to Stylized-ImageNet, Smooth-ImageNet and L0-ImageNet are shape-biased datasets and models trained on these datasets can produce effective models for shape-dominant data.

We proposed a basis for controlling the shape bias of CNNs using image decomposition as the dataset construction method; however, there is room for further improvement. We need to conduct evaluation experiments using datasets containing other images that are considered shape-dominant, such as medical images or billboards, to further demonstrate the effectiveness of creating datasets with image decomposition. Additionally, experiments on model robustness should be conducted. CNNs are known to misclassify even small noises, such as adversarial attacks. In contrast, the same type of adversarial attacks rarely causes misclassification among humans. Therefore, it is necessary to investigate the robustness of the model trained on the proposed dataset to simple noise, such as Gaussian noise, or adversarial attacks [3].

Although optimization methods were used for shape and texture image decomposition, they require parameters to be set for each image. The proposed dataset was created without changing the parameters for each image; therefore, it was not possible to erase the texture with the optimal parameters for all images. With over one million images in ImageNet, it is practically impossible to manually set parameters manually for each image, and creating an accurate dataset for larger datasets is still problematic. Therefore, to improve texture removal, a method that automatically sets parameters can be devised or an image generation model [14,39] that does not require parameter setting can be used.

Furthermore, it is necessary to devise an evaluation metric to measure the bias of the data to be fine-tuned. One advantage of the proposed method is that the ratio of global shape to local texture of the image can be varied by adding the decomposed images together again. By changing the ratio of shape and texture during training, a model with arbitrary bias can be created. In this study, we experimented with three types of shape-dominant images: animated images, sketches, and logo images. For example, when comparing a sketch image with an animated image, the animated image should have more local texture. Thus, we can expect more efficient learning and improved accuracy by creating a model with a bias tailored to the domain of each data set. For this purpose, it is necessary to have a measure of the bias of the data used for fine-tuning. This will enable the use of models with appropriate biases. In the future, the goal is to provide pretrained models with biases that match the biases of the fine-tuned data.

In this study, we created a dataset to train a CNN with a shape bias, which was fine-tuned and evaluated using data predominantly containing shapes. To achieve a shape bias, we employed an image decomposition method that separates images into their shape and texture components. This method overcomes the limitations of previous studies by preventing shape collapse during texture transformation, thus enabling us to reliably produce a shape-focused CNN. Our proposed method outperformed several baseline methods on test datasets where shape predominates. This dataset creation approach can be used to develop models with arbitrary biases by adjusting the proportions of shape and texture in the image through their combination. Moreover, the edges obtained through our method closely resemble those of the original images. By training on image datasets with varying proportions of shape and texture, CNNs with different biases can be created.

This work was supported in part by JSPS KAKENHI Grant Number 23K11174 and MEXT Promotion of Distinctive Joint Research Center Program Grant Number JPMXP 0621467946.