YOLOv10改进策略【卷积层】| ECCV-2024 Histogram Transformer 直方图自注意力 适用于噪声大，图像质量低的检测任务

一、本文介绍

本文记录的是利用直方图自注意力优化YOLOv10的目标检测方法研究。在目标检测任务中，清晰准确的图像对于目标检测至关重要，本文创新方法通过恢复图像质量，可以减少因图像质量低导致的误检和漏检，实现有效涨点。

---

专栏目录：YOLOv10改进目录一览 | 涉及卷积层、轻量化、注意力、损失函数、Backbone、SPPF、Neck、检测头等全方位改进

专栏地址：YOLOv10改进专栏——以发表论文的角度，快速准确的找到有效涨点的创新点！

---

二、直方图自注意力介绍

2.1 设计出发点

• 解决现有Transformer方法的局限：现有的基于Transformer的方法在处理恶劣天气图像恢复时，为了提高内存利用效率，通常将自注意力操作限制在固定的空间范围或仅仅在通道维度上，这种限制阻碍了Transformer对长距离空间特征的捕捉能力，从而影响了图像恢复的性能。
• 利用天气退化特征：观察到天气引起的退化因素主要导致相似的遮挡和亮度变化，因此希望设计一种能够更好地处理这些特征的模块。

2.2 原理

2.2.1 动态范围直方图自注意力（DHSA）

• 动态范围卷积：传统卷积操作的感受野范围有限，主要关注局部信息，与自注意力机制的长距离依赖建模能力不匹配。通过在传统卷积操作之前对输入特征进行重新排序，将其分为两个分支，对第一个分支的特征进行水平和垂直排序，然后与第二个分支的特征连接，再通过可分离卷积。这样可以将高强度和低强度的像素组织成矩阵对角线上的规则模式，使卷积能够在动态范围内进行计算，从而部分聚焦于保留干净信息和分别恢复退化特征。
• 直方图自注意力机制：注意到天气引起的退化会导致相似的模式，不同强度的包含背景特征或天气退化的像素应给予不同程度的注意力。因此提出将空间元素分类到不同的bin中，并在bin内和bin间分配不同的注意力。

2.2.2 双尺度门控前馈（DGFF）模块

• 考虑到之前的方法在标准前馈网络中通常使用单范围或单尺度卷积来增强局部上下文，但忽略了动态分布的天气引起的退化之间的相关性。因此设计了DGFF模块，它在传输过程中集成了两个不同的多范围和多尺度深度卷积路径，通过不同的卷积操作和门控机制来增强对多尺度和多范围信息的提取能力。

2.3 结构

• 包含两个关键模块
  • DHSA模块：由动态范围卷积和直方图自注意力机制组成。动态范围卷积对输入特征进行重新排序，直方图自注意力机制对重新排序后的特征进行处理，包括将特征分为Value特征和Query - Key对，对Value特征进行排序并根据其索引对Query - Key对进行排列，然后将特征重塑为两种类型（bin - wise直方图重塑和frequency - wise直方图重塑），分别通过两种重塑方式和后续的自注意力过程，最后将输出元素相乘得到最终输出。
  • DGFF模块：输入张量首先经过点卷积操作增加通道维度，然后分为两个并行分支。在特征转换过程中，一个分支使用5×5深度卷积，另一个分支使用扩张的3×3深度卷积来增强多范围和多尺度信息的提取。第二个分支的输出经过激活后作为门控图作用于第一个分支，最后通过像素重排和逆重排操作以及点卷积得到输出并传递到下一个阶段。

2.4 优势

• 有效捕捉动态范围的特征：DHSA模块通过动态范围卷积和直方图自注意力机制，能够有效地捕捉天气引起的动态空间退化特征，实现对长距离空间特征的建模，克服了现有方法的局限性。
• 提取多尺度和多范围信息：DGFF模块通过集成两个不同的多范围和多尺度深度卷积路径，能够更好地提取图像中的多尺度和多范围信息，增强了对天气退化图像的恢复能力。
• 提高图像恢复性能：通过上述两个模块的协同作用，Histogram Transformer Block能够提高恶劣天气图像恢复的性能，在多个数据集上取得了较好的效果。

论文：https://arxiv.org/pdf/2407.10172
源码：https://github.com/sunshangquan/Histoformer

三、HTB的实现代码

HTB模块的实现代码如下：

import numbers
import torch
import torch.nn as nn
import torch.nn.functional as F
from einops import rearrange
Conv2d = nn.Conv2d## Layer Norm
def to_2d(x):return rearrange(x, 'b c h w -> b (h w c)')def to_3d(x):
#    return rearrange(x, 'b c h w -> b c (h w)')return rearrange(x, 'b c h w -> b (h w) c')def to_4d(x,h,w):
#    return rearrange(x, 'b c (h w) -> b c h w',h=h,w=w)return rearrange(x, 'b (h w) c -> b c h w',h=h,w=w)class BiasFree_LayerNorm(nn.Module):def __init__(self, normalized_shape):super(BiasFree_LayerNorm, self).__init__()if isinstance(normalized_shape, numbers.Integral):normalized_shape = (normalized_shape,)normalized_shape = torch.Size(normalized_shape)assert len(normalized_shape) == 1self.normalized_shape = normalized_shapedef forward(self, x):sigma = x.var(-1, keepdim=True, unbiased=False)return x / torch.sqrt(sigma+1e-5) #* self.weightclass WithBias_LayerNorm(nn.Module):def __init__(self, normalized_shape):super(WithBias_LayerNorm, self).__init__()if isinstance(normalized_shape, numbers.Integral):normalized_shape = (normalized_shape,)normalized_shape = torch.Size(normalized_shape)assert len(normalized_shape) == 1self.normalized_shape = normalized_shapedef forward(self, x):mu = x.mean(-1, keepdim=True)sigma = x.var(-1, keepdim=True, unbiased=False)return (x - mu) / torch.sqrt(sigma+1e-5) #* self.weight + self.biasclass LayerNorm(nn.Module):def __init__(self, dim, LayerNorm_type="WithBias"):super(LayerNorm, self).__init__()if LayerNorm_type =='BiasFree':self.body = BiasFree_LayerNorm(dim)else:self.body = WithBias_LayerNorm(dim)def forward(self, x):h, w = x.shape[-2:]return to_4d(self.body(to_3d(x)), h, w)## Dual-scale Gated Feed-Forward Network (DGFF)
class FeedForward(nn.Module):def __init__(self, dim, ffn_expansion_factor, bias):super(FeedForward, self).__init__()hidden_features = int(dim * ffn_expansion_factor)self.project_in = Conv2d(dim, hidden_features * 2, kernel_size=1, bias=bias)self.dwconv_5 = Conv2d(hidden_features // 4, hidden_features // 4, kernel_size=5, stride=1, padding=2,groups=hidden_features // 4, bias=bias)self.dwconv_dilated2_1 = Conv2d(hidden_features // 4, hidden_features // 4, kernel_size=3, stride=1, padding=2,groups=hidden_features // 4, bias=bias, dilation=2)self.p_unshuffle = nn.PixelUnshuffle(2)self.p_shuffle = nn.PixelShuffle(2)self.project_out = Conv2d(hidden_features, dim, kernel_size=1, bias=bias)def forward(self, x):x = self.project_in(x)x = self.p_shuffle(x)x1, x2 = x.chunk(2, dim=1)x1 = self.dwconv_5(x1)x2 = self.dwconv_dilated2_1(x2)x = F.mish(x2) * x1x = self.p_unshuffle(x)x = self.project_out(x)return x##Dynamic-range Histogram Self-Attention (DHSA)
class Attention_histogram(nn.Module):def __init__(self, dim, num_heads=4, bias=False, ifBox=True):super(Attention_histogram, self).__init__()self.factor = num_headsself.ifBox = ifBoxself.num_heads = num_headsself.temperature = nn.Parameter(torch.ones(num_heads, 1, 1))self.qkv = Conv2d(dim, dim * 5, kernel_size=1, bias=bias)self.qkv_dwconv = Conv2d(dim * 5, dim * 5, kernel_size=3, stride=1, padding=1, groups=dim * 5, bias=bias)self.project_out = Conv2d(dim, dim, kernel_size=1, bias=bias)def pad(self, x, factor):hw = x.shape[-1]t_pad = [0, 0] if hw % factor == 0 else [0, (hw // factor + 1) * factor - hw]x = F.pad(x, t_pad, 'constant', 0)return x, t_paddef unpad(self, x, t_pad):_, _, hw = x.shapereturn x[:, :, t_pad[0]:hw - t_pad[1]]def softmax_1(self, x, dim=-1):logit = x.exp()logit = logit / (logit.sum(dim, keepdim=True) + 1)return logitdef normalize(self, x):mu = x.mean(-2, keepdim=True)sigma = x.var(-2, keepdim=True, unbiased=False)return (x - mu) / torch.sqrt(sigma + 1e-5)  # * self.weight + self.biasdef reshape_attn(self, q, k, v, ifBox):b, c = q.shape[:2]q, t_pad = self.pad(q, self.factor)k, t_pad = self.pad(k, self.factor)v, t_pad = self.pad(v, self.factor)hw = q.shape[-1] // self.factorshape_ori = "b (head c) (factor hw)" if ifBox else "b (head c) (hw factor)"shape_tar = "b head (c factor) hw"q = rearrange(q, '{} -> {}'.format(shape_ori, shape_tar), factor=self.factor, hw=hw, head=self.num_heads)k = rearrange(k, '{} -> {}'.format(shape_ori, shape_tar), factor=self.factor, hw=hw, head=self.num_heads)v = rearrange(v, '{} -> {}'.format(shape_ori, shape_tar), factor=self.factor, hw=hw, head=self.num_heads)q = torch.nn.functional.normalize(q, dim=-1)k = torch.nn.functional.normalize(k, dim=-1)attn = (q @ k.transpose(-2, -1)) * self.temperatureattn = self.softmax_1(attn, dim=-1)out = (attn @ v)out = rearrange(out, '{} -> {}'.format(shape_tar, shape_ori), factor=self.factor, hw=hw, b=b,head=self.num_heads)out = self.unpad(out, t_pad)return outdef forward(self, x):b, c, h, w = x.shapex_sort, idx_h = x[:, :c // 2].sort(-2)x_sort, idx_w = x_sort.sort(-1)x[:, :c // 2] = x_sortqkv = self.qkv_dwconv(self.qkv(x))q1, k1, q2, k2, v = qkv.chunk(5, dim=1)  # b,c,x,xv, idx = v.view(b, c, -1).sort(dim=-1)q1 = torch.gather(q1.view(b, c, -1), dim=2, index=idx)k1 = torch.gather(k1.view(b, c, -1), dim=2, index=idx)q2 = torch.gather(q2.view(b, c, -1), dim=2, index=idx)k2 = torch.gather(k2.view(b, c, -1), dim=2, index=idx)out1 = self.reshape_attn(q1, k1, v, True)out2 = self.reshape_attn(q2, k2, v, False)out1 = torch.scatter(out1, 2, idx, out1).view(b, c, h, w)out2 = torch.scatter(out2, 2, idx, out2).view(b, c, h, w)out = out1 * out2out = self.project_out(out)out_replace = out[:, :c // 2]out_replace = torch.scatter(out_replace, -1, idx_w, out_replace)out_replace = torch.scatter(out_replace, -2, idx_h, out_replace)out[:, :c // 2] = out_replacereturn out##Histogram Transformer Block (HTB)
class HTB(nn.Module):def __init__(self, dim, num_heads=1, ffn_expansion_factor=2.5, bias=False, LayerNorm_type='WithBias'):## Other option 'BiasFree'super(HTB, self).__init__()self.attn_g = Attention_histogram(dim, num_heads, bias, True)self.norm_g = LayerNorm(dim, LayerNorm_type)self.ffn = FeedForward(dim, ffn_expansion_factor, bias)self.norm_ff1 = LayerNorm(dim, LayerNorm_type)def forward(self, x):x = x + self.attn_g(self.norm_g(x))x_out = x + self.ffn(self.norm_ff1(x))return x_out

---

四、创新模块

4.1 改进点⭐

模块改进方法：直接加入HTB（第五节讲解添加步骤）。

HTB模块加入如下：

4.2 改进点⭐

模块改进方法：基于HTB模块的C2fCIB（第五节讲解添加步骤）。

第二种改进方法是对YOLOv10中的C2fCIB模块进行改进，并将HTB在加入到C2fCIB模块中。

改进代码如下：

首先对CIB模块进行改进，加入HTB模块

class CIB(nn.Module):"""Standard bottleneck."""def __init__(self, c1, c2, shortcut=True, e=0.5, lk=False):"""Initializes a bottleneck module with given input/output channels, shortcut option, group, kernels, andexpansion."""super().__init__()c_ = int(c2 * e)  # hidden channelsself.cv1 = nn.Sequential(Conv(c1, c1, 3, g=c1),Conv(c1, 2 * c_, 1),Conv(2 * c_, 2 * c_, 3, g=2 * c_) if not lk else RepVGGDW(2 * c_),Conv(2 * c_, c2, 1),Conv(c2, c2, 3, g=c2),HTB(c2))self.add = shortcut and c1 == c2def forward(self, x):"""'forward()' applies the YOLO FPN to input data."""return x + self.cv1(x) if self.add else self.cv1(x)

然后，将C2fCIB重命名为C2fCIB_HTB

class C2fCIB_HTB(C2f):"""Faster Implementation of CSP Bottleneck with 2 convolutions."""def __init__(self, c1, c2, n=1, shortcut=False, lk=False, g=1, e=0.5):"""Initialize CSP bottleneck layer with two convolutions with arguments ch_in, ch_out, number, shortcut, groups,expansion."""super().__init__(c1, c2, n, shortcut, g, e)self.m = nn.ModuleList(CIB(self.c, self.c, shortcut, e=1.0, lk=lk) for _ in range(n))

注意❗：在第五小节中需要声明的模块名称为：HTB和C2fCIB_HTB。

---

五、添加步骤

5.1 修改一

① 在ultralytics/nn/目录下新建AddModules文件夹用于存放模块代码

② 在AddModules文件夹下新建HTB.py，将第三节中的代码粘贴到此处

5.2 修改二

在AddModules文件夹下新建__init__.py（已有则不用新建），在文件内导入模块：from .HTB import *

5.3 修改三

在ultralytics/nn/modules/tasks.py文件中，需要在两处位置添加各模块类名称。

首先：导入模块

其次：在parse_model函数中注册HTB和C2fCIB_HTB模块

---

六、yaml模型文件

6.1 模型改进版本1⭐

此处以ultralytics/cfg/models/v10/yolov10m.yaml为例，在同目录下创建一个用于自己数据集训练的模型文件yolov10m-HTB.yaml。

将yolov10m.yaml中的内容复制到yolov10m-HTB.yaml文件下，修改nc数量等于自己数据中目标的数量。

📌 模型的修改方法是将骨干网络中的C2f和C2fCIB模块替换成HTB模块。

# Parameters
nc: 1 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov8n.yaml' will call yolov8.yaml with scale 'n'# [depth, width, max_channels]m: [0.67, 0.75, 768] # YOLOv8m summary: 295 layers, 25902640 parameters, 25902624 gradients,  79.3 GFLOPsbackbone:# [from, repeats, module, args]- [-1, 1, Conv, [64, 3, 2]] # 0-P1/2- [-1, 1, Conv, [128, 3, 2]] # 1-P2/4- [-1, 3, HTB, [128]]- [-1, 1, Conv, [256, 3, 2]] # 3-P3/8- [-1, 6, HTB, [256]]- [-1, 1, SCDown, [512, 3, 2]] # 5-P4/16- [-1, 6, HTB, [512]]- [-1, 1, SCDown, [1024, 3, 2]] # 7-P5/32- [-1, 3, HTB, [1024]]- [-1, 1, SPPF, [1024, 5]] # 9- [-1, 1, PSA, [1024]] # 10# YOLOv8.0n head
head:- [-1, 1, nn.Upsample, [None, 2, "nearest"]]- [[-1, 6], 1, Concat, [1]] # cat backbone P4- [-1, 3, C2f, [512]] # 13- [-1, 1, nn.Upsample, [None, 2, "nearest"]]- [[-1, 4], 1, Concat, [1]] # cat backbone P3- [-1, 3, C2f, [256]] # 16 (P3/8-small)- [-1, 1, Conv, [256, 3, 2]]- [[-1, 13], 1, Concat, [1]] # cat head P4- [-1, 3, C2f, [512]] # 19 (P4/16-medium)- [-1, 1, SCDown, [512, 3, 2]]- [[-1, 10], 1, Concat, [1]] # cat head P5- [-1, 3, C2fCIB, [1024, True, True]] # 22 (P5/32-large)- [[16, 19, 22], 1, v10Detect, [nc]] # Detect(P3, P4, P5)

6.2 模型改进版本2⭐

此处以ultralytics/cfg/models/v10/yolov10m.yaml为例，在同目录下创建一个用于自己数据集训练的模型文件yolov10m-C3k2_HTB.yaml。

将yolov10m.yaml中的内容复制到yolov10m-C3k2_HTB.yaml文件下，修改nc数量等于自己数据中目标的数量。

📌 模型的修改方法是将骨干网络中的C2fCIB模块替换成C3k2_HTB模块。

# Parameters
nc: 1 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov8n.yaml' will call yolov8.yaml with scale 'n'# [depth, width, max_channels]m: [0.67, 0.75, 768] # YOLOv8m summary: 295 layers, 25902640 parameters, 25902624 gradients,  79.3 GFLOPsbackbone:# [from, repeats, module, args]- [-1, 1, Conv, [64, 3, 2]] # 0-P1/2- [-1, 1, Conv, [128, 3, 2]] # 1-P2/4- [-1, 3, C2f, [128, True]]- [-1, 1, Conv, [256, 3, 2]] # 3-P3/8- [-1, 6, C2f, [256, True]]- [-1, 1, SCDown, [512, 3, 2]] # 5-P4/16- [-1, 6, C2f, [512, True]]- [-1, 1, SCDown, [1024, 3, 2]] # 7-P5/32- [-1, 3, C2fCIB_HTB, [1024, True, True]]- [-1, 1, SPPF, [1024, 5]] # 9- [-1, 1, PSA, [1024]] # 10# YOLOv8.0n head
head:- [-1, 1, nn.Upsample, [None, 2, "nearest"]]- [[-1, 6], 1, Concat, [1]] # cat backbone P4- [-1, 3, C2f, [512]] # 13- [-1, 1, nn.Upsample, [None, 2, "nearest"]]- [[-1, 4], 1, Concat, [1]] # cat backbone P3- [-1, 3, C2f, [256]] # 16 (P3/8-small)- [-1, 1, Conv, [256, 3, 2]]- [[-1, 13], 1, Concat, [1]] # cat head P4- [-1, 3, C2f, [512]] # 19 (P4/16-medium)- [-1, 1, SCDown, [512, 3, 2]]- [[-1, 10], 1, Concat, [1]] # cat head P5- [-1, 3, C2fCIB, [1024, True, True]] # 22 (P5/32-large)- [[16, 19, 22], 1, v10Detect, [nc]] # Detect(P3, P4, P5)

---

七、成功运行结果

打印网络模型可以看到HTB和C2fCIB_HTB已经加入到模型中，并可以进行训练了。

YOLOv10m-HTB：

YOLOv10m-HTB summary: 552 layers, 31,100,662 parameters, 31,100,646 gradients, 113.5 GFLOPs

                   from  n    params  module                                       arguments                     0                  -1  1      1392  ultralytics.nn.modules.conv.Conv             [3, 48, 3, 2]                 1                  -1  1     41664  ultralytics.nn.modules.conv.Conv             [48, 96, 3, 2]                2                  -1  2    261744  ultralytics.nn.AddModules.HTB.HTB            [96, 96]                      3                  -1  1    166272  ultralytics.nn.modules.conv.Conv             [96, 192, 3, 2]               4                  -1  4   2042304  ultralytics.nn.AddModules.HTB.HTB            [192, 192]                    5                  -1  1     78720  ultralytics.nn.modules.block.SCDown          [192, 384, 3, 2]              6                  -1  4   8065920  ultralytics.nn.AddModules.HTB.HTB            [384, 384]                    7                  -1  1    228672  ultralytics.nn.modules.block.SCDown          [384, 576, 3, 2]              8                  -1  2   9035424  ultralytics.nn.AddModules.HTB.HTB            [576, 576]                    9                  -1  1    831168  ultralytics.nn.modules.block.SPPF            [576, 576, 5]                 10                  -1  1   1253088  ultralytics.nn.modules.block.PSA             [576, 576]                    11                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          12             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           13                  -1  2   1993728  ultralytics.nn.modules.block.C2f             [960, 384, 2]                 14                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          15             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           16                  -1  2    517632  ultralytics.nn.modules.block.C2f             [576, 192, 2]                 17                  -1  1    332160  ultralytics.nn.modules.conv.Conv             [192, 192, 3, 2]              18            [-1, 13]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           19                  -1  2   1846272  ultralytics.nn.modules.block.C2f             [576, 384, 2]                 20                  -1  1    152448  ultralytics.nn.modules.block.SCDown          [384, 384, 3, 2]              21            [-1, 10]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           22                  -1  2   1969920  ultralytics.nn.modules.block.C2fCIB          [960, 576, 2, True, True]     23        [16, 19, 22]  1   2282134  ultralytics.nn.modules.head.v10Detect        [1, [192, 384, 576]]          
YOLOv10m-HTB summary: 552 layers, 31,100,662 parameters, 31,100,646 gradients, 113.5 GFLOPs

**YOLOv10m-C2fCIB_HTB **：

YOLOv10m-C2fCIB_HTB summary: 534 layers, 20,502,072 parameters, 20,502,056 gradients, 69.6 GFLOPs

                   from  n    params  module                                       arguments                     0                  -1  1      1392  ultralytics.nn.modules.conv.Conv             [3, 48, 3, 2]                 1                  -1  1     41664  ultralytics.nn.modules.conv.Conv             [48, 96, 3, 2]                2                  -1  2    111360  ultralytics.nn.modules.block.C2f             [96, 96, 2, True]             3                  -1  1    166272  ultralytics.nn.modules.conv.Conv             [96, 192, 3, 2]               4                  -1  4    813312  ultralytics.nn.modules.block.C2f             [192, 192, 4, True]           5                  -1  1     78720  ultralytics.nn.modules.block.SCDown          [192, 384, 3, 2]              6                  -1  4   3248640  ultralytics.nn.modules.block.C2f             [384, 384, 4, True]           7                  -1  1    228672  ultralytics.nn.modules.block.SCDown          [384, 576, 3, 2]              8                  -1  2   4633490  ultralytics.nn.AddModules.HTB.C2fCIB_HTB     [576, 576, True, True]        9                  -1  1    831168  ultralytics.nn.modules.block.SPPF            [576, 576, 5]                 10                  -1  1   1253088  ultralytics.nn.modules.block.PSA             [576, 576]                    11                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          12             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           13                  -1  2   1993728  ultralytics.nn.modules.block.C2f             [960, 384, 2]                 14                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          15             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           16                  -1  2    517632  ultralytics.nn.modules.block.C2f             [576, 192, 2]                 17                  -1  1    332160  ultralytics.nn.modules.conv.Conv             [192, 192, 3, 2]              18            [-1, 13]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           19                  -1  2   1846272  ultralytics.nn.modules.block.C2f             [576, 384, 2]                 20                  -1  1    152448  ultralytics.nn.modules.block.SCDown          [384, 384, 3, 2]              21            [-1, 10]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           22                  -1  2   1969920  ultralytics.nn.modules.block.C2fCIB          [960, 576, 2, True, True]     23        [16, 19, 22]  1   2282134  ultralytics.nn.modules.head.v10Detect        [1, [192, 384, 576]]          
YOLOv10m-C2fCIB_HTB summary: 534 layers, 20,502,072 parameters, 20,502,056 gradients, 69.6 GFLOPs