電子發(fā)燒友網(wǎng)>電子資料下載>電子資料>PyTorch教程6.4之惰性初始化

PyTorch教程6.4之惰性初始化

2078771 2023-06-05 | pdf | 0.11 MB | 次下載 | 免費(fèi)

資料介紹

到目前為止，似乎我們?cè)诮?a target='_blank' class='arckwlink_none'>網(wǎng)絡(luò)時(shí)草率地逃脫了懲罰。具體來(lái)說(shuō)，我們做了以下不符合直覺(jué)的事情，這些事情可能看起來(lái)不應(yīng)該起作用：

我們?cè)跊](méi)有指定輸入維度的情況下定義了網(wǎng)絡(luò)架構(gòu)。
我們添加層時(shí)沒(méi)有指定前一層的輸出維度。
在提供足夠的信息來(lái)確定我們的模型應(yīng)該包含多少參數(shù)之前，我們甚至“初始化”了這些參數(shù)。

您可能會(huì)對(duì)我們的代碼完全運(yùn)行感到驚訝。畢竟，深度學(xué)習(xí)框架無(wú)法判斷網(wǎng)絡(luò)的輸入維數(shù)。這里的技巧是框架推遲初始化，等到我們第一次通過(guò)模型傳遞數(shù)據(jù)時(shí)，動(dòng)態(tài)推斷每一層的大小。

稍后，當(dāng)使用卷積神經(jīng)網(wǎng)絡(luò)時(shí)，該技術(shù)將變得更加方便，因?yàn)檩斎刖S度（即圖像的分辨率）將影響每個(gè)后續(xù)層的維度。因此，在編寫(xiě)代碼時(shí)無(wú)需知道維度是多少就可以設(shè)置參數(shù)的能力可以極大地簡(jiǎn)化指定和隨后修改模型的任務(wù)。接下來(lái)，我們將更深入地研究初始化機(jī)制。

						import torch
from torch import nn
from d2l import torch as d2l

						 

						from mxnet import np, npx
from mxnet.gluon import nn

npx.set_np()

						import jax
from flax import linen as nn
from jax import numpy as jnp
from d2l import jax as d2l

						 

						No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

					

						import tensorflow as tf

						 

首先，讓我們實(shí)例化一個(gè) MLP。

						net = nn.Sequential(nn.LazyLinear(256), nn.ReLU(), nn.LazyLinear(10))

						 

						net = nn.Sequential()
net.add(nn.Dense(256, activation='relu'))
net.add(nn.Dense(10))

						 

						net = nn.Sequential([nn.Dense(256), nn.relu, nn.Dense(10)])

						 

						net = tf.keras.models.Sequential([
  tf.keras.layers.Dense(256, activation=tf.nn.relu),
  tf.keras.layers.Dense(10),
])

						 

此時(shí)，網(wǎng)絡(luò)不可能知道輸入層權(quán)重的維度，因?yàn)檩斎刖S度仍然未知。

因此框架還沒(méi)有初始化任何參數(shù)。我們通過(guò)嘗試訪問(wèn)以下參數(shù)來(lái)確認(rèn)。

						net[0].weight

						 

						<UninitializedParameter>

					

Consequently the framework has not yet initialized any parameters. We confirm by attempting to access the parameters below.

						print(net.collect_params)
print(net.collect_params())

						<bound method Block.collect_params of Sequential(
 (0): Dense(-1 -> 256, Activation(relu))
 (1): Dense(-1 -> 10, linear)
)>
sequential0_ (
 Parameter dense0_weight (shape=(256, -1), dtype=float32)
 Parameter dense0_bias (shape=(256,), dtype=float32)
 Parameter dense1_weight (shape=(10, -1), dtype=float32)
 Parameter dense1_bias (shape=(10,), dtype=float32)
)

					

Note that while the parameter objects exist, the input dimension to each layer is listed as -1. MXNet uses the special value -1 to indicate that the parameter dimension remains unknown. At this point, attempts to access net[0].weight.data() would trigger a runtime error stating that the network must be initialized before the parameters can be accessed. Now let’s see what happens when we attempt to initialize parameters via the initialize method.

						net.initialize()
net.collect_params()

						sequential0_ (
 Parameter dense0_weight (shape=(256, -1), dtype=float32)
 Parameter dense0_bias (shape=(256,), dtype=float32)
 Parameter dense1_weight (shape=(10, -1), dtype=float32)
 Parameter dense1_bias (shape=(10,), dtype=float32)
)

					

As we can see, nothing has changed. When input dimensions are unknown, calls to initialize do not truly initialize the parameters. Instead, this call registers to MXNet that we wish (and optionally, according to which distribution) to initialize the parameters.

As mentioned in Section 6.2.1, parameters and the network definition are decoupled in Jax and Flax, and the user handles both manually. Flax models are stateless hence there is no parameters attribute.

Consequently the framework has not yet initialized any parameters. We confirm by attempting to access the parameters below.

						[net.layers[i].get_weights() for i in range(len(net.layers))]

						 

[[], []]

Note that each layer objects exist but the weights are empty. Using net.get_weights() would throw an error since the weights have not been initialized yet.

接下來(lái)讓我們通過(guò)網(wǎng)絡(luò)傳遞數(shù)據(jù)，讓框架最終初始化參數(shù)。

						X = torch.rand(2, 20)
net(X)

net[0].weight.shape

						torch.Size([256, 20])

					

						X = np.random.uniform(size=(2, 20))
net(X)

net.collect_params()

						sequential0_ (
 Parameter dense0_weight (shape=(256, 20), dtype=float32)
 Parameter dense0_bias (shape=(256,), dtype=float32)
 Parameter dense1_weight (shape=(10, 256), dtype=float32)
 Parameter dense1_bias (shape=(10,), dtype=float32)
)

					

						params = net.init(d2l.get_key(), jnp.zeros((2, 20)))
jax.tree_util.tree_map(lambda x: x.shape, params).tree_flatten()

						(({'params': {'layers_0': {'bias': (256,), 'kernel': (20, 256)},
  'layers_2': {'bias': (10,), 'kernel': (256, 10)}}},),
 ())

					

						X = tf.random.uniform((2, 20))
net(X)
[w.shape for w in net.get_weights()]

						 

						[(20, 256), (256,), (256, 10), (10,)]

					

一旦我們知道輸入維度 20，框架就可以通過(guò)插入值 20 來(lái)識(shí)別第一層權(quán)重矩陣的形狀。識(shí)別出第一層的形狀后，框架進(jìn)入第二層，依此類(lèi)推計(jì)算圖，直到所有形狀都已知。請(qǐng)注意，在這種情況下，只有第一層需要延遲初始化，但框架會(huì)按順序進(jìn)行初始化。一旦知道所有參數(shù)形狀，框架就可以最終初始化參數(shù)。

以下方法通過(guò)網(wǎng)絡(luò)傳入虛擬輸入以進(jìn)行試運(yùn)行，以推斷所有參數(shù)形狀并隨后初始化參數(shù)。當(dāng)不需要默認(rèn)隨機(jī)初始化時(shí)，稍后將使用它。

						@d2l.add_to_class(d2l.Module) #@save
def apply_init(self, inputs, init=None):
  self.forward(*inputs)
  if init is not None:
    self.net.apply(init)

						 

Parameter initialization in Flax is always done manually and handled by the user. The following method takes a dummy input and a key dictionary as argument. This key dictionary has the rngs for initializing the model parameters and dropout rng for generating the dropout mask for the models with dropout layers. More about dropout will be covered later in Section 5.6. Ultimately the method initializes the model returning the parameters. We have been using it under the hood in the previous sections as well.

						@d2l.add_to_class(d2l.Module) #@save
def apply_init(self, dummy_input, key):
  params = self.init(key, *dummy_input) # dummy_input tuple unpacked
  return params

						 