U 0Jdd:d;Z?dd?fd@dAZCddBdCZDdDdEZEedFdGfedH<edIdJfedK<ddLdMZFdNdOZGdPdQZHddRdSZIdTdUZJGdVdWdWe jZKe*jdXdYfe*jdZ<d[d\ZLdd^d_ZMGd`dadaZNdbdcZOddedfZPdgdhZQdidjZRGdkdldlZSddodpZTGdqdrdrZUdsdtZVdudvZWdwdxZXddydzZYGd{d|d|e jZZe*jd}d~fe*jd<ddZ[ddZ\dddZ]ddZ^ddZ_ddZ`dddZaGddde jZbGddde jZcGddde jZdGddde*jbZedddZfGddde jgZhddZidddZjddZkdddZlddZmGddde jZnGddde jZoGddde*jcZpGddde jZqddZrddZsGddde jZtGddde jZuGddde jZvGddde jZwGddde*jbZxddZydddZzddÄZ{e*jddfe*jd<dddʄZ|ddd̈́Z}dddЄZ~Gdd҄d҃ZddԄZdddׄZddلZe*GfddۄZe*jddfe*jd<ddZddZddZddZdddZddZdddZdddZddZddZddZdddZe*jddfe*jd<dddZGdddejjjZddZe*jddfe*jd<dddZdddgdddgdddgdddgdddgdddgdddgdddgd ddgd ddgd ddgd ddgd ddgd ddgd ddgd ddgdd dgdd dgdd dgdd dgdd dggZd d d ddddddddddddddddddgZd d!Zd"d#Zd$d%ZGd&d'd'ejjjZd(d)Ze*jd*d+fe*jd,<d-d.Zd/d0Zd1d2Zd3d4Ze*jd5d6fe*jd7<e*jd8d9fe*jd:<d;d<Zd=d>Zd?d@ZGdAdBdBZdCdDZdEdFZdGdHZe*jdIdJfe*jdK<e*jdLdMfe*jdN<e*jdOdPfe*jdQ<e*jdRdSfe*jdT<GdUdVdVZddWdXZGdYdZdZe jZGd[d\d\e jZGd]d^d^e jZGd_d`d`e jZdae*jdb<dcddZdedfZdgdhZdidjZdkdlZdmdnZGdodpdpejjjZdqdrZdsdtZdue*jdv<dwdxZddzd{Zd|d}Zd~e*jd<ddZGdddejjjZdddZddZddZddZe*jddfe*jd<ejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZejZ֐ddZddZאddZؐddZِddZڐddZېddZܐddZdS(z,http://d2l-data.s3-accelerate.amazonaws.com/N)Image)nn) functionaldata) transforms) defaultdict)display)pyplot)backend_inlinecCstddS)zWUse the svg format to display a plot in Jupyter. Defined in :numref:`sec_calculus`svgN)r set_matplotlib_formatsrr8/home/d2l-worker/workspace/d2l-en-release/./d2l/torch.pyuse_svg_display1srg @@cCst|tjjd<dS)zJSet the figure size for matplotlib. Defined in :numref:`sec_calculus`zfigure.figsizeN)rd2lpltrcParamsfigsizerrr set_figsize7srcCsV|||||||||||||rJ|||dS)zCSet the axes for matplotlib. Defined in :numref:`sec_calculus`N) set_xlabel set_ylabel set_xscale set_yscaleset_xlimset_ylimlegendgrid)axesxlabelylabelxlimylimxscaleyscalerrrrset_axes>s       r(linear-zm--zg-.zr:c  Cs|dkr g}t| | r| ntj} dd} | |r<|g}|dkrZggt||}}n| |rh|g}t|t|kr|t|}| t||| D].\} }}t| r| | ||q| ||qt| |||||||dS)z8Plot data points. Defined in :numref:`sec_calculus`NcSs.t|dr|jdkp,t|to,t|dd S)Nndimr__len__)hasattrr, isinstancelistXrrr has_one_axisYszplot..has_one_axis) rrrgcalenclazipplotr()r3Yr"r#rr$r%r&r'fmtsrr!r4xyfmtrrrr9Ls& r9c@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)TimerzRecord multiple running times.cCsg|_|dS)z(Defined in :numref:`subsec_linear_model`N)timesstartselfrrr__init__oszTimer.__init__cCst|_dS)zStart the timer.N)timetikrBrrrrAtsz Timer.startcCs |jt|j|jdS)z-Stop the timer and record the time in a list.)r@appendrErFrBrrrstopxsz Timer.stopcCst|jt|jS)zReturn the average time.)sumr@r6rBrrravg}sz Timer.avgcCs t|jS)zReturn the sum of time.)rJr@rBrrrrJsz Timer.sumcCst|jS)zReturn the accumulated time.)nparrayr@cumsumtolistrBrrrrNsz Timer.cumsumN) __name__ __module__ __qualname____doc__rDrArIrKrJrNrrrrr?msr?cCsJtdd|t|f}t|||}|tdd|j7}|t|dfS)zIGenerate y = Xw + b + noise. Defined in :numref:`sec_linear_scratch`rr-{Gz?rGr-)rnormalr6matmulshapereshape)wb num_examplesr3r=rrrsynthetic_datasr]cCst|||S)zIThe linear regression model. Defined in :numref:`sec_linear_scratch`)rrW)r3rZr[rrrlinregsr^cCs|t||jddS)z:Squared loss. Defined in :numref:`sec_linear_scratch`)rrYrX)y_hatr=rrr squared_losssrac Cs>t,|D] }|||j|8}|jqW5QRXdS)zSMinibatch stochastic gradient descent. Defined in :numref:`sec_linear_scratch`N)torchno_gradgradZzero_)paramslr batch_sizeparamrrrsgds riTcCstj|}tj|||dS)zOConstruct a PyTorch data iterator. Defined in :numref:`sec_linear_concise`shuffle)rZ TensorDataset DataLoader) data_arraysrgis_traindatasetrrr load_arrays rpc s*ddddddddd d g fd d |DS) z]Return text labels for the Fashion-MNIST dataset. Defined in :numref:`sec_fashion_mnist`zt-shirttrouserpulloverdresscoatsandalshirtsneakerbagz ankle bootcsg|]}t|qSr)int.0i text_labelsrr sz,get_fashion_mnist_labels..r)labelsrr}rget_fashion_mnist_labelss r?c Cs||||f}tjj|||d\}}|}tt||D]b\}\} } t| rb| | n | | | j d| j d|r<| ||q<|S)zBPlot a list of images. Defined in :numref:`sec_fashion_mnist`rF)rrsubplotsflatten enumerater8rb is_tensorimshownumpyr! get_xaxis set_visible get_yaxis set_title) imgsnum_rowsnum_colstitlesscaler_r!r|aximgrrr show_imagess  rcCsdS)zMUse 4 processes to read the data. Defined in :numref:`sec_fashion_mnist`rrrrrget_dataloader_workerssrcCsztg}|r |dt|t|}tjjdd|dd}tjjdd|dd}tj ||dt dtj ||dt dfS)zlDownload the Fashion-MNIST dataset and then load it into memory. Defined in :numref:`sec_fashion_mnist`rz../dataT)roottrain transformdownloadFrk num_workers) rToTensorinsertResizeCompose torchvisiondatasets FashionMNISTrrlr)rgresizetrans mnist_train mnist_testrrrload_data_fashion_mnists.    rcCsTt|jdkr*|jddkr*tj|dd}t||j|k}ttt||jS)zXCompute the number of correct predictions. Defined in :numref:`sec_softmax_scratch`r-axis)r6rXrargmaxastypedtypefloat reduce_sum)r`r=cmprrraccuracysrc Cslt|tjjr|td}t0|D]$\}}|t|||t |q,W5QRX|d|dS)z\Compute the accuracy for a model on a dataset. Defined in :numref:`sec_softmax_scratch`r_rr-) r0rbrModuleeval Accumulatorrcaddrrsize)net data_itermetricr3r=rrrevaluate_accuracys  (rc@s0eZdZdZddZddZddZdd Zd S) rz)For accumulating sums over `n` variables.cCsdg||_dS)(Defined in :numref:`sec_softmax_scratch`Nr)rCnrrrrDszAccumulator.__init__cGsddt|j|D|_dS)NcSsg|]\}}|t|qSrr)r{ar[rrrrsz#Accumulator.add..)r8rrCargsrrrrszAccumulator.addcCsdgt|j|_dS)Nr)r6rrBrrrresetszAccumulator.resetcCs |j|SNrrCidxrrr __getitem__szAccumulator.__getitem__N)rPrQrRrSrDrrrrrrrrs rc Cst|tjjr|td}|D]\}}||}|||}t|tjjrh|| | n| ||j d|t| t|||q"|d|d|d|dfS)zUThe training loop defined in Chapter 3. Defined in :numref:`sec_softmax_scratch`rr_r-)r0rbrrrroptim Optimizer zero_gradmeanbackwardsteprJrXrrrnumel) r train_iterlossupdaterrr3r=r`lrrrtrain_epoch_ch3s     "rc @s"eZdZdZd ddZd d ZdS) AnimatorzFor plotting data in animation.Nr)r*r-rc szdkr gttjj| | | d\__| | dkrDjg_fdd_dd|___ dS)rNrr-c stjdSNr)rr(r!rrrCr"r$r&r#r%r'rr/sz#Animator.__init__..) rrrrfigr! config_axesr3r:r;) rCr"r#rr$r%r&r'r;nrowsncolsrrrrrD"s  zAnimator.__init__cCst|ds|g}t|}t|ds,|g|}|jsFddt|D|_|js`ddt|D|_tt||D]<\}\}}|dk rn|dk rn|j|||j||qn|jd t|j|j|j D]\}}}|jd |||q| t |jt jdddS)Nr.cSsg|]}gqSrrr{rrrrr;sz Animator.add..cSsg|]}gqSrrrrrrr=srT)wait)r/r6r3ranger:rr8rHr!r7r;r9rr r clear_output)rCr<r=rr|rr[r>rrrr3s&    z Animator.add) NNNNNr)r)r*r-r-r)rPrQrRrSrDrrrrrr s rc Cstdd|gddgdddgd}t|D]2}t||||}t||} ||d|| fq&|\} } | d ksrt| | dkr| d kst| | dkr| d kst| d S) zSTrain a model (defined in Chapter 3). Defined in :numref:`sec_softmax_scratch`epochr-g333333?g? train loss train acctest accr"r$r%r?gffffff?N)rrrrrAssertionError) rr test_iterr num_epochsranimatorr train_metricstest_acc train_loss train_accrrr train_ch3Is  rcCs||D] \}}qqt|}ttj||dd}ddt||D}tjt|d||ddfd||d|ddS) zTPredict labels (defined in Chapter 3). Defined in :numref:`sec_softmax_scratch`r-rcSsg|]\}}|d|qS) r)r{truepredrrrr`szpredict_ch3..r)rN)rrrr8rrY)rrrr3r=truespredsrrrr predict_ch3Xs   rcCs`td}|D]@\}}||}t||j}|||}|t|t|q|d|dS)z`Evaluate the loss of a model on the given dataset. Defined in :numref:`sec_model_selection`r_rr-)rrrYrXrrr)rrrrr3r=outrrrr evaluate_lossds   rz..rc Cs|tkst|dtdt|\}}tj|ddtj||dd}tj|rt }t |d"}| d}|s~q| |qnW5QRX| |kr|Std |d |d tj|ddd }t |d }||jW5QRX|S)zmDownload a file inserted into DATA_HUB, return the local filename. Defined in :numref:`sec_kaggle_house`z does not exist in .Texist_ok/rGrbiz Downloading z from z...)streamverifywb)DATA_HUBrosmakedirspathjoinsplitexistshashlibsha1openreadupdate hexdigestprintrequestsgetwritecontent) name cache_dirurl sha1_hashfnamer frrrrrrss$      rcCs|t|}tj|}tj|\}}|dkr:t|d}n"|dkrPt|d}n ds\t d| ||rxtj ||S|S)zODownload and extract a zip/tar file. Defined in :numref:`sec_kaggle_house`z.zipr)z.tarz.gzFz$Only zip/tar files can be extracted.) rrrdirnamesplitextzipfileZipFiletarfiler r extractallr)rfolderrbase_dirdata_dirextfprrrdownload_extracts   r&cCstD] }t|qdS)zNDownload all files in the DATA_HUB. Defined in :numref:`sec_kaggle_house`N)rr)rrrr download_allsr'zkaggle_house_pred_train.csv(585e9cc93e70b39160e7921475f9bcd7d31219cekaggle_house_trainzkaggle_house_pred_test.csv(fa19780a7b011d9b009e8bff8e99922a8ee2eb90kaggle_house_testcCs,tj|dkr"td|StdS)zVReturn gpu(i) if exists, otherwise return cpu(). Defined in :numref:`sec_use_gpu`r-cuda:cpu)rbcuda device_countdevice)r|rrrtry_gpusr1cCs,ddttjD}|r |StdgS)z^Return all available GPUs, or [cpu(),] if no GPU exists. Defined in :numref:`sec_use_gpu`cSsg|]}td|qS)r,)rbr0rzrrrrsz try_all_gpus..r-)rrbr.r/r0devicesrrr try_all_gpuss r4c Cs|j\}}t|jd|d|jd|df}t|jdD]F}t|jdD]2}t|||||||f||||f<qTqB|S)zFCompute 2D cross-correlation. Defined in :numref:`sec_conv_layer`rr-)rXrzerosrr)r3KhrZr:r|jrrrcorr2ds  *2r9c st|tjr*|s*tt|jt d}t d|D]X\}}t|t rhfdd|D}n | }| }|t|||t|qBW5QRX|d|dS)z^Compute the accuracy for a model on a dataset using a GPU. Defined in :numref:`sec_lenet`r_csg|]}|qSrtor{r<r0rrrsz)evaluate_accuracy_gpu..rr-)r0rrrnextiter parametersr0rrrbrcr1r;rrr)rrr0rr3r=rr=revaluate_accuracy_gpus       *rAc Csdd}||td|||tjj||d}t}t j dd|gddd gd } t t |} } t |D]6} t d } |t|D]\}\}}| |||||}}||}|||}||t,| ||jd t |||jd W5QRX| | d | d }| d| d }|d| dd ksn|| dkr| | |d| ||dfqt||}| | ddd|fqvtd|dd|dd|dt| d || ddt|dS)zTTrain a model with a GPU (defined in Chapter 6). Defined in :numref:`sec_lenet`cSs.t|tjkst|tjkr*tj|jdSr)typerLinearConv2dinitxavier_uniform_weightmrrr init_weightssztrain_ch6..init_weightsz training onrfrr-rrr)r"r$rrrr_Nloss .3f , train acc , test acc .1f examples/sec on )applyrr;rbrSGDr@rCrossEntropyLossrrr?r6rrrrrArrrrcrrXrrIrArJstr)rrrrrfr0rJ optimizerrrtimer num_batchesrrr|r3r=r`rtrain_lrrrrr train_ch6sD       0"  r[cs*eZdZdZdfdd ZddZZS) ResidualzThe Residual block of ResNet.Fr-cspttj||dd|d|_tj||ddd|_|rNtj||d|d|_nd|_t||_t||_ dS)Nrr-) kernel_sizepaddingstride)r]r^)r]r_) superrDrrDconv1conv2conv3 BatchNorm2dbn1bn2)rCZinput_channels num_channels use_1x1convstrides __class__rrrDs$   zResidual.__init__cCsHt|||}|||}|jr6||}||7}t|Sr)FrelurerarfrbrcrCr3r:rrrforwards  zResidual.forward)Fr-rPrQrRrSrDro __classcell__rrrjrr\s r\ztimemachine.txt(090b5e7e70c295757f55df93cb0a180b9691891a time_machinec Cs2ttdd}|}W5QRXdd|DS)ziLoad the time machine dataset into a list of text lines. Defined in :numref:`sec_text_preprocessing`rsrcSs"g|]}tdd|qS)z [^A-Za-z]+ )resubstriplowerr{linerrrr'sz%read_time_machine..)r rr readlines)rlinesrrrread_time_machine!sr}wordcCs<|dkrdd|DS|dkr,dd|DStd|dS)z`Split text lines into word or character tokens. Defined in :numref:`sec_text_preprocessing`r~cSsg|] }|qSrrryrrrr.sztokenize..charcSsg|] }t|qSr)r1ryrrrr0szERROR: unknown token type: N)r)r|tokenrrrtokenize)s rc@sJeZdZdZdddZddZdd Zd d Zed d Z eddZ dS)VocabzVocabulary for text.NrcCs|dkr g}|dkrg}t|}t|dddd|_dg||_ddt|jD|_|jD]>\}}||krtq||jkr`|j|t|jd |j|<q`dS) z+Defined in :numref:`sec_text_preprocessing`NcSs|dSNr-rr<rrrr>z Vocab.__init__..T)keyreversecSsi|]\}}||qSrrr{rrrrr Bsz"Vocab.__init__..r-) count_corpussorteditems _token_freqs idx_to_tokenr token_to_idxrHr6)rCtokensmin_freqreserved_tokenscounterrfreqrrrrD6s$   zVocab.__init__cCs t|jSrr6rrBrrrr.Ksz Vocab.__len__cs0t|ttfsj|jSfdd|DS)Ncsg|]}|qSr)rr{rrBrrrQsz%Vocab.__getitem__..)r0r1tuplerrunk)rCrrrBrrNszVocab.__getitem__cs*t|ttfsj|Sfdd|DS)Ncsg|]}j|qSrr)r{indexrBrrrVsz#Vocab.to_tokens..)r0r1rr)rCindicesrrBr to_tokensSs zVocab.to_tokenscCsdSrrrBrrrrXsz Vocab.unkcCs|jSr)rrBrrr token_freqs\szVocab.token_freqs)NrN) rPrQrRrSrDr.rrpropertyrrrrrrr4s  rcCs2t|dkst|dtr(dd|D}t|S)zICount token frequencies. Defined in :numref:`sec_text_preprocessing`rcSsg|]}|D]}|q qSrrr{rzrrrrrgsz count_corpus..)r6r0r1 collectionsCounter)rrrrr`srrGcsFt}t|d}t|fdd|D}|dkr>|d|}|fS)zuReturn token indices and the vocabulary of the time machine dataset. Defined in :numref:`sec_text_preprocessing`rcsg|]}|D] }|q qSrrrvocabrrrssz,load_corpus_time_machine..rN)r}rr) max_tokensr|rcorpusrrrload_corpus_time_machinejs  rc #stdddtd}ttd|}t|fdd||}td|||D]N}||||}fdd|D}fdd|D} t|t| fVqldS) zhGenerate a minibatch of subsequences using random sampling. Defined in :numref:`sec_language_model`rr-Ncs||Srr)pos)r num_stepsrrrsz"seq_data_iter_random..datacsg|] }|qSrrr{r8rrrrsz(seq_data_iter_random..csg|]}|dqSr-rrrrrrs)randomrandintr6r1rrkrtensor) rrgr num_subseqsinitial_indicesrYr|initial_indices_per_batchr3r:r)rrrrseq_data_iter_randomxs rc cstd|}t||d||}t||||}t||d|d|}||d||d}}|jd|}td|||D]>}|dd|||f} |dd|||f} | | fVqdS)zpGenerate a minibatch of subsequences using sequential partitioning. Defined in :numref:`sec_language_model`rr-rGN)rrr6rrrYrXr) rrgroffset num_tokensXsYsrYr|r3r:rrrseq_data_iter_sequentials rc@s eZdZdZddZddZdS) SeqDataLoaderz"An iterator to load sequence data.cCs:|rtj|_ntj|_t|\|_|_|||_|_dS)z'Defined in :numref:`sec_language_model`N) rr data_iter_fnrrrrrgr)rCrgruse_random_iterrrrrrDs  zSeqDataLoader.__init__cCs||j|j|jSr)rrrgrrBrrr__iter__szSeqDataLoader.__iter__N)rPrQrRrSrDrrrrrrs rF'cCst||||}||jfS)zpReturn the iterator and the vocabulary of the time machine dataset. Defined in :numref:`sec_language_model`)rr)rgrrrrrrrload_data_time_machinesrc@s(eZdZdZddZddZddZdS) RNNModelScratchz%A RNN Model implemented from scratch.cCs.|||_|_|||||_|||_|_dS)z$Defined in :numref:`sec_rnn_scratch`N) vocab_size num_hiddensre init_state forward_fn)rCrrr0 get_paramsrrrrrrDszRNNModelScratch.__init__cCs(t|j|jtj}||||jSr) rlone_hotTrrBrbfloat32rrerCr3staterrr__call__szRNNModelScratch.__call__cCs|||j|Sr)rr)rCrgr0rrr begin_stateszRNNModelScratch.begin_stateN)rPrQrRrSrDrrrrrrrsrc s|jdd}|dgfdd}|ddD]"}|||\}}|q6t|D]0}|||\}}t|jdddqbdfd d DS) zYGenerate new characters following the `prefix`. Defined in :numref:`sec_rnn_scratch`r-rgr0rcsttjdgddS)NrGr=r-r-)rrYrr)r0outputsrrrs zpredict_ch8..Ndimcsg|]}j|qSrrrzrrrrszpredict_ch8..)rrHrryrrYr) prefix num_predsrrr0r get_inputr=rr)r0rrr predict_ch8s rcCsnt|tjr dd|D}n|j}ttdd|D}||krj|D]}|jdd||9<qJdS)z.css|]}t|jdVqdS)r_N)rbrJrdrrrr sz grad_clipping..N) r0rrr@rerbsqrtrJrd)rthetarenormrhrrr grad_clippings rcCsNdt}}td}|D]\} } |dks2|rH|j| jd|d}n2t|tjrht|tsh| n|D] } | ql| j d} | || |} } || |\} }|| | }t|tjjr||t|d|n|t|d|dd||t| t| qt|d|d|d|fS)z^Train a net within one epoch (defined in Chapter 8). Defined in :numref:`sec_rnn_scratch`Nr_rrrGr-rg)rr?rrrXr0rrrZdetach_rrYr;longrrbrrrrrrrrmathexprI)rrrrr0rrrXrr3r:sr=r`rrrrtrain_epoch_ch8s.         rcst}tjdddgd|gd}ttjr@tj } nfdd} fdd} t |D]H} t ||| |\} } | d dd krft | d | | d | gqft d | d d| d dtt | d t | ddS)zOTrain a model (defined in Chapter 8). Defined in :numref:`sec_rnn_scratch`r perplexityr )r"r#rr$cstj|Sr)rrirer)rfrrrrrztrain_ch8..cst|dS)N2)r)r)r0rrrrrrr-rztime travellerz perplexity rQ,  tokens/sec on travellerN)rrUrrr0rrbrrTr@rrrrrV)rrrrfrr0rrrrpredictrpplspeedr)r0rfrrr train_ch8s0   " rcs2eZdZdZfddZddZd ddZZS) RNNModelz8The RNN model. Defined in :numref:`sec_rnn-concise`c sjtt|jf|||_||_|jj|_|jjsJd|_t |j|j|_ nd|_t |jd|j|_ dS)Nr-r_) r`rrDrnnr hidden_sizer bidirectionalnum_directionsrrCr))rC rnn_layerrkwargsrjrrrD+s zRNNModel.__init__cCsRt|j|j}|tj}|||\}}| | d|j df}||fS)NrG) rlrrrrr;rbrrr)rYrX)rCinputsrr3r:outputrrrro9s  zRNNModel.forwardr-cCsrt|jtjs.tj|j|jj||jf|dStj|j|jj||jf|dtj|j|jj||jf|dfSdS)Nr=) r0rrLSTMrbr5r num_layersr)rCr0rgrrrrCs,  zRNNModel.begin_state)r-)rPrQrRrSrDrorrqrrrjrr's  rz fra-eng.zip(94646ad1522d915e7b0f9296181140edcf86a4f5fra-engc Cs>td}ttj|dd}|W5QRSQRXdS)zRLoad the English-French dataset. Defined in :numref:`sec_machine_translation`rzfra.txtrN)rr&r rrrr )r#rrrr read_data_nmtUs rcsBddddddfddtD}d|S) zXPreprocess the English-French dataset. Defined in :numref:`sec_machine_translation`cSs|tdko|dkS)Nz,.!?rt)set)r prev_charrrrno_spaceasz preprocess_nmt..no_spaceu rt cs6g|].\}}|dkr.||dr.d|n|qS)rr-rtr)r{r|rrtextrrrhsz"preprocess_nmt..r)replacerxrr)rrrrrpreprocess_nmt]s  rcCsxgg}}t|dD]V\}}|r0||kr0qp|d}t|dkr||dd||ddq||fS)zVTokenize the English-French dataset. Defined in :numref:`sec_machine_translation`r r_rrtr-)rrr6rH)rr\sourcetargetr|rzpartsrrr tokenize_nmtls    r cCstttjdd|Ddd|Dg\}}}tj|tj||djD]}|dqTtj|dS)z[Plot the histogram for list length pairs. Defined in :numref:`sec_machine_translation`cSsg|] }t|qSrr6r{rrrrrsz+show_list_len_pair_hist..r-rN) rrrhistr"r#patches set_hatchr)rr"r#xlistylistrr patchrrrshow_list_len_pair_histzs    rcCs.t||kr|d|S||g|t|S)zLTruncate or pad sequences. Defined in :numref:`sec_machine_translation`Nr )rzr padding_tokenrrr truncate_pads  rcsdfdd|D}fdd|D}tfdd|D}tt|dktjd}||fS)zrTransform text sequences of machine translation into minibatches. Defined in :numref:`subsec_mt_data_loading`csg|] }|qSrrr rrrrsz#build_array_nmt..csg|]}|dgqS)rr rrrrscsg|]}t|dqS)rr rrrrrs rr-)rrrrint32)r|rrrM valid_lenrrrbuild_array_nmtsrXcCstt}t||\}}tj|ddddgd}tj|ddddgd}t|||\}} t|||\} } || | | f} t| |} | ||fS)zuReturn the iterator and the vocabularies of the translation dataset. Defined in :numref:`subsec_mt_data_loading`r_rrrr)rrr rrrrp)rgrr\rrr src_vocab tgt_vocab src_array src_valid_len tgt_array tgt_valid_lenrmrrrr load_data_nmts   r%cs(eZdZdZfddZddZZS)Encoderz@The base encoder interface for the encoder-decoder architecture.c stt|jf|dSr)r`r&rDrCrrjrrrDszEncoder.__init__cGstdSrNotImplementedError)rCr3rrrrroszEncoder.forwardrprrrjrr&s r&cs0eZdZdZfddZddZddZZS)DecoderznThe base decoder interface for the encoder-decoder architecture. Defined in :numref:`sec_encoder-decoder`c stt|jf|dSr)r`r*rDr'rjrrrDszDecoder.__init__cGstdSrr()rC enc_outputsrrrrrszDecoder.init_statecCstdSrr(rrrrroszDecoder.forward)rPrQrRrSrDrrorqrrrjrr*s r*cs(eZdZdZfddZddZZS)EncoderDecoderzbThe base class for the encoder-decoder architecture. Defined in :numref:`sec_encoder-decoder`c s"tt|jf|||_||_dSr)r`r,rDencoderdecoder)rCr-r.rrjrrrDszEncoderDecoder.__init__cGs.|j|f|}|jj|f|}|||Sr)r-r.r)rCenc_Xdec_Xrr+ dec_staterrrroszEncoderDecoder.forwardrprrrjrr,s r,cs*eZdZdZdfdd ZddZZS)Seq2SeqEncoderzXThe RNN encoder for sequence to sequence learning. Defined in :numref:`sec_seq2seq`rc s8tt|jf|t|||_tj||||d|_dS)N)dropout)r`r2rDr Embedding embeddingGRUr)rCr embed_sizerrr3rrjrrrDs  zSeq2SeqEncoder.__init__cGs.||}|ddd}||\}}||fS)Nr-rr_)r5permuter)rCr3rrrrrrros zSeq2SeqEncoder.forward)rrprrrjrr2sr2cCsH|d}tj|tj|jddddf|dddfk}|||<|S)zSMask irrelevant entries in sequences. Defined in :numref:`sec_seq2seq_decoder`r-rr0N)rrbarangerr0)r3rvaluemaxlenmaskrrr sequence_masks    r>cs eZdZdZfddZZS)MaskedSoftmaxCELosszXThe softmax cross-entropy loss with masks. Defined in :numref:`sec_seq2seq_decoder`csJt|}t||}d|_tt||ddd|}||jdd}|S)Nnonerr_r-r) rb ones_liker> reductionr`r?ror8r)rCrlabelrweightsZunweighted_lossZ weighted_lossrjrrros    zMaskedSoftmaxCELoss.forward)rPrQrRrSrorqrrrjrr?sr?c sdd}|||tjj||d}t}|tj ddd|gd} t |D]} t } t d} |D]} | fd d | D\}}}}tj|d g|jd d dd}t||ddddfgd}||||\}}||||}|t|d|}|t| ||W5QRXqv| ddd krZ| | d| d | dfqZtd| d | ddd| d| ddtdS)zUTrain a model for sequence to sequence. Defined in :numref:`sec_seq2seq_decoder`cSsTt|tjkrtj|jt|tjkrP|jD]}d|kr0tj|j|q0dS)NrG) rBrrCrErFrGr6Z_flat_weights_names _parameters)rIrhrrrxavier_init_weightss  z*train_seq2seq..xavier_init_weightsrKrrr)r"r#r$r_csg|]}|qSrr:r<r=rrrsz!train_seq2seq..rrr=rGr-NrMrNrrQr)rSr;rbrAdamr@r?rrrrr?rrrrXrYconcatrJrrrrcrrrIrV)rrrfrr r0rFrWrrrrXrbatchr3 X_valid_lenr: Y_valid_lenbos dec_inputY_hatrrrrr=r train_seq2seqsB         rOcCs4|||d|dg}tjt|g|d}t|||d}tjtj|tj |ddd} | | |} |j | |} tjtj|dgtj |ddd} gg} }t |D]j}| | | \}} |jd d} | jddtj}|r||j j||dkrq | |qd|| |fS) zPPredict for sequence to sequence. Defined in :numref:`sec_seq2seq_training`rtrr=rr9rrrr_)rrxrrbrr6rr unsqueezerr-r.rrrsqueezerBritemrHattention_weightsrr)r src_sentencerr rr0save_attention_weights src_tokens enc_valid_lenr/r+r1r0 output_seqattention_weight_seqrr:rrrrpredict_seq2seq's<     rZc Cs"|d|d}}t|t|}}ttdd||}td|dD]}dtt} } t||dD]&} | d || | |d7<qrt||dD]L} | d || | |dkr| d7} | d || | |d8<q|t | ||dt d|9}qN|S)z@Compute the BLEU. Defined in :numref:`sec_seq2seq_training`rtrr-r) rr6rrminrrrryrpow) pred_seq label_seqk pred_tokens label_tokenslen_pred len_labelscorer num_matches label_subsr|rrrbleuKs$$&rgrrRedscCst|jd|jd}}tjj|||dddd\}} tt| |D]v\} \} } tt| | D]Z\} \}}|jt||d}| |dkr| || dkr| ||rb| || qbqH|j || ddd S) zGShow heatmaps of matrices. Defined in :numref:`sec_attention-cues`rr-TF)rsharexshareyrQ)cmapg333333?)rshrinkN) rrrXrrrr8rrrrrcolorbar)matricesr"r#rrrlrrrr!r|row_axes row_matricesr8rmatrixpcmrrr show_heatmaps]s"     rtcCs||dkrtjj|ddS|j}|dkr)r3 valid_lensrXrrrmasked_softmaxqs  rxcs(eZdZdZfddZddZZS)AdditiveAttentionzMAdditive attention. Defined in :numref:`sec_attention-scoring-functions`c sXtt|jf|tj||dd|_tj||dd|_tj|ddd|_t||_ dS)NFbiasr-) r`ryrDrrCW_kW_qw_vDropoutr3)rCkey_size query_sizerr3rrjrrrDs zAdditiveAttention.__init__cCsd||||}}|d|d}t|}||d}t|||_t | |j|S)Nr_r-rG) r}r|rPrbtanhr~rQrxrSbmmr3)rCquerieskeysvaluesrwfeaturesscoresrrrros   zAdditiveAttention.forwardrprrrjrrys rycs*eZdZdZfddZdddZZS)DotProductAttentionzQScaled dot product attention. Defined in :numref:`subsec_additive-attention`c s"tt|jf|t||_dSr)r`rrDrrr3)rCr3rrjrrrDszDotProductAttention.__init__NcCsH|jd}t||ddt|}t|||_t||j|S)NrGr-r_) rXrbr transposerrrxrSr3)rCrrrrwdrrrrros  zDotProductAttention.forward)Nrprrrjrrs rcs,eZdZdZfddZeddZZS)AttentionDecoderz[The base attention-based decoder interface. Defined in :numref:`sec_seq2seq_attention`c stt|jf|dSr)r`rrDr'rjrrrDszAttentionDecoder.__init__cCstdSrr(rBrrrrSsz"AttentionDecoder.attention_weights)rPrQrRrSrDrrSrqrrrjrrs rcs*eZdZdZdfdd ZddZZS)MultiHeadAttentionzGMulti-head attention. Defined in :numref:`sec_multihead-attention`Fc  sptt|jf|||_t||_tj|||d|_ tj|||d|_ tj|||d|_ tj|||d|_ dS)Nrz) r`rrD num_headsrr attentionrrCr}r|W_vW_o) rCrr value_sizerrr3r{rrjrrrDs zMultiHeadAttention.__init__cCsvt|||j}t|||j}t|||j}|dk rPtj||jdd}|||||}t||j}| |S)Nr)repeatsr) transpose_qkvr}rr|rrbrvrtranspose_outputr)rCrrrrwr output_concatrrrros zMultiHeadAttention.forward)Frprrrjrrs rcCsF||jd|jd|d}|dddd}|d|jd|jdS)zuTransposition for parallel computation of multiple attention heads. Defined in :numref:`sec_multihead-attention`rr-rGr_rrYrXr8r3rrrrrs rcCsF|d||jd|jd}|dddd}||jd|jddS)z[Reverse the operation of `transpose_qkv`. Defined in :numref:`sec_multihead-attention`rGr-r_rrrrrrrrsrcs*eZdZdZdfdd ZddZZS)PositionalEncodingzYPositional encoding. Defined in :numref:`sec_self-attention-and-positional-encoding`c stt|t||_td||f|_tj |t j d ddt dt j d|dt j d|}t ||jdddddddf<t ||jdddddddf<dS)Nr-rrGrrr_)r`rrDrrr3rr5Pr:rbrrYr\sincos)rCrr3max_lenr3rjrrrD s$  $zPositionalEncoding.__init__cCs8||jddd|jdddf|j}||Sr)rrXr;r0r3rCr3rrrros.zPositionalEncoding.forward)rrprrrjrrs rcs(eZdZdZfddZddZZS)PositionWiseFFNzLPositionwise feed-forward network. Defined in :numref:`sec_transformer`c s<tt|jf|t|||_t|_t|||_dSr) r`rrDrrCdense1ReLUrmdense2)rC ffn_num_inputffn_num_hiddensffn_num_outputsrrjrrrDs zPositionWiseFFN.__init__cCs||||Sr)rrmrrrrrro$szPositionWiseFFN.forwardrprrrjrrs rcs(eZdZdZfddZddZZS)AddNormz^Residual connection followed by layer normalization. Defined in :numref:`sec_transformer`c s.tt|jf|t||_t||_dSr)r`rrDrrr3 LayerNormln)rCZnormalized_shaper3rrjrrrD+s zAddNorm.__init__cCs||||Sr)rr3rnrrrro0szAddNorm.forwardrprrrjrr's rcs*eZdZdZdfdd ZddZZS) EncoderBlockzDTransformer encoder block. Defined in :numref:`sec_transformer`Fc sTtt|jf| t|||||| | |_t|| |_t||||_ t|| |_ dSr) r`rrDrrrraddnorm1rffnaddnorm2) rCrrrr norm_shaperrrr3use_biasrrjrrrD7s" zEncoderBlock.__init__c Cs*|||||||}||||Sr)rrrr)rCr3rwr:rrrroCszEncoderBlock.forward)Frprrrjrr3s rcs*eZdZdZdfdd ZddZZS)TransformerEncoderz>Transformer encoder. Defined in :numref:`sec_transformer`Fc  s|tt|jf| ||_t|||_t|| |_ t |_ t | D]0}|j dt|t|||||||| | | qFdS)Nblock)r`rrDrrr4r5rr pos_encoding Sequentialblksr add_modulerVr)rCrrrrrrrrrrr3rrr|rjrrrDKs    zTransformerEncoder.__init__cGs`|||t|j}dgt|j|_t|jD]"\}}|||}|j j j|j|<q8|Sr) rr5rrrr6rrSrr)rCr3rwrr|blkrrrroYs zTransformerEncoder.forward)FrprrrjrrGsrcCs"tjj|||tddddS)Nz->) arrowstyle)xyxytext arrowprops)rrr5annotatedict)rrrrrrresrc Csd\}}}}||fg}t|D]F}|r@||||||\}}}}n|||||\}}}}|||fqtd|ddt|ddt|d|S)zlOptimize a 2D objective function with a customized trainer. Defined in :numref:`subsec_gd-learningrate`)rrzepoch r-z, x1: rz, x2: )rrHrr) trainerstepsf_gradx1x2s1s2resultsr|rrrtrain_2dis   *rcCs|ttjjt|d ddittdddtddd\}}tjj|||||dd tjd tj d d S)zdShow the trace of 2D variables during optimization. Defined in :numref:`subsec_gd-learningrate`-ocolorz#ff7f0eg?g?gz#1f77b4)colorsrrN)r) rrrr9r8meshgridr:contourr"r#)rrrrrrr show_trace_2dys  rzairfoil_self_noise.dat(76e5be1548fd8222e5074cf0faae75edff8cf93fairfoilrcCs|tjtdtjdd}t||jdd|jdd}tj |d|ddf|d|dff|dd }||j d d fS) $Defined in :numref:`sec_minibatches`rr)r delimiterrrNrGTrnr-) rL genfromtxtrrrrb from_numpyrstdrprX)rgrrrrrr get_data_ch11s "&rr_c s6tjdd|dfddtjdddfddtj}}tjd d d |gd d gd}d t} } t|D]} |D]\} } ||| | }| |g||| | j d 7} | dd krr| | | | j d t |t|||f| qrqjtd|jd ddd| dd| |jd fS)rrrTr-T)rrrrrcst|Sr)rr^r2r[rZrrrrztrain_ch11..rrr)\(?ffffff?r"r#r$r%loss: rGrNr sec/epoch)rbrVr5rrarr?rrrrXrIrr6rrArr:rKrN) trainer_fnstates hyperparamsr feature_dimrrrrrrXrr3r=rrrr train_ch11s0    (rrc CsBttdd}dd}||||f|}tjdd}tjddd |gd d gd }d t} } t |D]} |D]\} } | || }| |j } ||| }| || | j d 7} | d d krx| || | j d t|t|||df| qxqptd|jd ddd| dddS)rrLr-cSs&t|tjkr"tjjj|jdddS)NrT)r)rBrrCrbrEZnormal_rGrHrrrrJsz(train_concise_ch11..init_weightsr@)rBrrrrrrrr_rrGrNrrN)rrrCrSr@ZMSELossrrr?rrrYrXrrrrIrr6rrArr:rK)rrrrrrJrWrrrrXrr3r=rrrrrtrain_concise_ch11s4         rc@s*eZdZdZd ddZddZddZd S) BenchmarkzFor measuring running time.DonecCs ||_dS)z"Defined in :numref:`sec_hybridize`N) description)rCrrrrrDszBenchmark.__init__cCst|_|Sr)rr?rXrBrrr __enter__s zBenchmark.__enter__cGs"t|jd|jdddS)Nz: z.4fz sec)rrrXrIrrrr__exit__szBenchmark.__exit__N)r)rPrQrRrSrDrrrrrrrs rcCs4|jd|jdksttj||tj||fS)zPSplit `X` and `y` into multiple devices. Defined in :numref:`sec_multi_gpu`r)rXrrparallelscatter)r3r=r3rrr split_batchs  rr-c Csddd}ttj|dddddtdt}|d|ddd d d |d |dd d |d|d dd |d|ddd |dtd|dtttd||S)zTA slightly modified ResNet-18 model. Defined in :numref:`sec_multi_gpu_concise`Fc SsTg}t|D]<}|dkr6|s6|tj||dddq |t||q tj|S)NrTr_)rhri)rrHrr\rr) in_channelsZ out_channels num_residuals first_blockrr|rrr resnet_blocks    zresnet18..resnet_block@rr-)r]r_r^Z resnet_block1r_T)rZ resnet_block2Z resnet_block3Z resnet_block4iZglobal_avg_poolrfc)F) rrrDrdrrZAdaptiveAvgPool2dFlattenrC) num_classesrrrrrrresnet18s   rc st|trfdd|D}n|d}|d}||||}|||}|||}t ||} || fS)zqTrain for a minibatch with mutiple GPUs (defined in Chapter 13). Defined in :numref:`sec_image_augmentation`csg|]}|dqSrr:r<r2rrrsz$train_batch_ch13..r) r0r1r;rrrJrrrr) rr3r=rrr3rrtrain_loss_sum train_acc_sumrr2rtrain_batch_ch13s    rc Cstt|}}tjdd|gddgdddgd} tj||d|d}t|D]} td } t |D]\} \} }| t || ||||\}}| |||j d||| d|d dks| |dkrh| | | d|| d| d | d| d d fqht||}| | dd d |fqRtd| d| d dd| d| d dd|dt| d ||ddt|d S)ziTrain a model with mutiple GPUs (defined in Chapter 13). Defined in :numref:`sec_image_augmentation`rr-rrrrr)Z device_idsrrLr_rNrMrNrOrPrQrR)rr?r6rrZ DataParallelr;rrrrArrrXrrIrArrJrV)rrrrrrr3rXrYrrrr|rrraccrrrr train_ch13s:    8rz hotdog.zip(fba480ffa8aa7e0febbb511d181409f899b9baa5hotdogc Cs|dddf|dddf|dddf|dddff\}}}}||d}||d}||}||}tj||||fdd}|S)zeConvert from (upper-left, lower-right) to (center, width, height). Defined in :numref:`sec_bbox`Nrr-r_rrGrrstack) boxesry1ry2cxcyrZr7rrrbox_corner_to_center2sD  r c Cs|dddf|dddf|dddf|dddff\}}}}|d|}|d|}|d|}|d|}tj||||fdd}|S) zeConvert from (center, width, height) to (upper-left, lower-right). Defined in :numref:`sec_bbox`Nrr-r_rrrGrr) rrr rZr7rrrrrrrbox_center_to_corner>sD    r cCs<tjj|d|df|d|d|d|dd|ddS)zMConvert bounding box to matplotlib format. Defined in :numref:`sec_bbox`rr-r_rF)rwidthheightfill edgecolor linewidth)rr Rectangle)bboxrrrr bbox_to_rectJsrc Cs|jdd\}}|jt|t|}}}||d}tj||d} tj||d} d\} } d|} d|}tj||d| | }tj||d| |}t||\}}|d|d}}t | t | d|dt | ddf||}t | t | d|dt | ddf}t | | ||fj ||dd }tj ||||gdd j|dd }||}|dS) zhGenerate anchor boxes with different shapes centered on each pixel. Defined in :numref:`sec_anchor`rNr-r=)rrrrGrr_r)rXr0r6rrrbr:rrYcatrrrrepeatrvrP)rsizesratios in_heightin_widthr0 num_sizes num_ratiosboxes_per_pixel size_tensor ratio_tensoroffset_hoffset_wsteps_hsteps_wcenter_hcenter_wshift_yshift_xrZr7anchor_manipulationsout_gridrrrrmultibox_priorUsH r)c Csddd}||}||dddddg}t|D]\}}||t|}tt||}|||r.t||kr.|d kr~d nd } |j|jd |jd ||d d d| t|d ddq.dS)z9Show bounding boxes. Defined in :numref:`sec_anchor`NcSs&|dkr|}nt|ttfs"|g}|Sr)r0r1r)objdefault_valuesrrr make_lists zshow_bboxes..make_listr[grrIcrZr_rr-center ) facecolorlw)vahafontsizerr)N) rr6rrr add_patchrrr) r!bboxesrrr,r|rrrect text_colorrrr show_bboxes|s    r:c Csdd}||}||}t|dddddf|ddddf}t|dddddf|ddddf}||jdd}|dddddf|dddddf}|dddf||} || S)zgCompute pairwise IoU across two lists of anchor or bounding boxes. Defined in :numref:`sec_anchor`cSs@|dddf|dddf|dddf|dddfS)Nr_rrr-r)rrrrrszbox_iou..Nr_r)r[r-)rbmaxr[clamp) boxes1boxes2box_areaareas1areas2inter_upperleftsinter_lowerrightsinters inter_areas union_areasrrrbox_ious..,rGrcCs|jd|jd}}t||}tj|fdtj|d}tj|dd\}} t|dkd} | |dk} | || <t|fd} t|fd} t|D]N}t |}||}||}|||<| |dd|f<| ||ddf<q|S)z`Assign closest ground-truth bounding boxes to anchor boxes. Defined in :numref:`sec_anchor`rrGr9r-rrN) rXrGrbfullrr;nonzerorYrr) ground_truthanchorsr0 iou_threshold num_anchors num_gt_boxesjaccardanchors_bbox_mapmax_iousranc_ibox_j col_discard row_discardrmax_idxbox_idxanc_idxrrrassign_anchor_to_bboxs&      rYư>c Cst|}t|}d|ddddf|ddddf|ddddf}dt||ddddf|ddddf}tj||gdd}|S)zXTransform for anchor box offsets. Defined in :numref:`subsec_labeling-anchor-boxes`rNr_rLr-r)rr logrH)rK assigned_bbepsc_anc c_assigned_bb offset_xy offset_whrrrr offset_boxess   @6rbcCs\|jd|d}}ggg}}}|j|jd}}t|D]}||ddddf} t| ddddf||} | dkddd} tj |tj |d} tj |dftj |d} t | dk}| |}| |df d| |<| |ddf| |<t || | }||d|| d|| q@t|}t|} t|} || | fS)zlLabel anchor boxes using ground-truth bounding boxes. Defined in :numref:`subsec_labeling-anchor-boxes`rNr-rGrr9)rXrQr0rrYrrPrrbr5rrrIrbrHrYr)rKrrg batch_offset batch_maskbatch_class_labelsr0rMr|rCrP bbox_mask class_labelsr\ indices_truebb_idxr bbox_offsetrrrmultibox_targets@      rkcCst|}|ddddf|ddddfd|ddddf}t|ddddfd|ddddf}tj||fdd}t|}|S)z{Predict bounding boxes based on anchor boxes with predicted offsets. Defined in :numref:`subsec_labeling-anchor-boxes`Nr_rrLr-r)rr rrHr )rK offset_predsanc pred_bbox_xy pred_bbox_wh pred_bboxpredicted_bboxrrroffset_inverses  @2 rrcCstj|ddd}g}|dkr|d}|||dkr@qt||ddfdd||ddddfddd}t||kd}||d}qtj||j dS) zrSort confidence scores of predicted bounding boxes. Defined in :numref:`subsec_predicting-bounding-boxes-nms`rGT)rZ descendingrr-Nrr=) rbargsortrrHrGrYrIrrr0)rrrLBkeepr|iouindsrrrnmss   rxQ%z?cCsN|j|jd}}|d}|jd|jd}}g} t|D]} || || dd} } t| ddd\} }t|| }t|| |}tj |tj |d}t ||f}|j dd \}}||dk}t ||f}d||<||}| |||} }| |k}d||<d| || |<tj | d| d|fdd }| |q>t| S) ztPredict bounding boxes using non-maximum suppression. Defined in :numref:`subsec_predicting-bounding-boxes-nms`rr-r_rGrNr9T) return_countsr)r0rXrQrrYrbr;rrrxr:rruniquerPrHrr) cls_probsrlrK nms_threshold pos_thresholdr0rgrrMrr|cls_prob offset_predconfclass_id predicted_bbruall_idxcombineduniquescountsnon_keep all_id_sorted below_min_idx pred_inforrrmultibox_detections8      rzbanana-detection.zip(5de26c8fce5ccdea9f91267273464dc968d20d72banana-detectionc Cstd}tj||rdndd}t|}|d}gg}}|D]@\}}| t j tj||rjdndd|| t |qH|t|ddfS) zkRead the banana detection dataset images and labels. Defined in :numref:`sec_object-detection-dataset`r bananas_train bananas_valz label.csvimg_nameimagesr-r)rr&rrrpdread_csv set_indexiterrowsrHrio read_imager1rbrrP)rnr# csv_fnamecsv_datartargetsrrrrrread_data_bananas4s$     rc@s(eZdZdZddZddZddZdS) BananasDatasetzqA customized dataset to load the banana detection dataset. Defined in :numref:`sec_object-detection-dataset`cCs6t|\|_|_tdtt|j|r*dnddS)Nread z training examplesz validation examples)rrrrrVr6)rCrnrrrrDLs zBananasDataset.__init__cCs|j||j|fSr)rrrrrrrrQszBananasDataset.__getitem__cCs t|jSrr6rrBrrrr.TszBananasDataset.__len__NrPrQrRrSrDrr.rrrrrHsrcCs8tjjjtdd|dd}tjjtdd|}||fS)zYLoad the banana detection dataset. Defined in :numref:`sec_object-detection-dataset`TrrjF)rbutilsrrlr)rgrval_iterrrrload_data_bananasWsrzVOCtrainval_11-May-2012.tar(4e443f8a2eca6b1dac8a6c57641b67dd40621a49voc2012c Cstj|dd|rdnd}tjjjj}t|d}| }W5QRXgg}}t |D]R\}} | tj tj|d| d| tj tj|d| d |qZ||fS) zZRead all VOC feature and label images. Defined in :numref:`sec_semantic_segmentation` ImageSets Segmentationz train.txtzval.txtr JPEGImagesz.jpgSegmentationClassz.png)rrrrrimageZ ImageReadModeRGBr r rrrHr) voc_dirrn txt_fnamemoderrrrr|rrrrread_voc_imagesds*     rrr background aeroplanebicyclebirdboatbottlebuscarrchaircow diningtabledoghorse motorbikepersonz potted plantsheepsofarz tv/monitorcCsJtjdtjd}ttD],\}}|||dd|dd|d<q|S)zoBuild the mapping from RGB to class indices for VOC labels. Defined in :numref:`sec_semantic_segmentation`irrrr-r_)rbr5rr VOC_COLORMAP)colormap2labelr|colormaprrrvoc_colormap2labelsrcCsj|dddd}|dddddfd|dddddfd|dddddf}||S)zlMap any RGB values in VOC labels to their class indices. Defined in :numref:`sec_semantic_segmentation`r-r_rrNr)r8rr)rrrrrrvoc_label_indicess 2rcCsDtjj|||f}tjjj|f|}tjjj|f|}||fS)z`Randomly crop both feature and label images. Defined in :numref:`sec_semantic_segmentation`)rrZ RandomCroprrcrop)featurerCr r r8rrr voc_rand_cropsrc@s8eZdZdZddZddZddZdd Zd d Zd S) VOCSegDatasetzaA customized dataset to load the VOC dataset. Defined in :numref:`sec_semantic_segmentation`cstjjdddgdddgd_|_t||d\}}fd d |D_|_t _ t d t t jd dS) Ng ףp= ?gv/?gCl?gZd;O?gy&1?g?)rrrcsg|]}|qSr)normalize_image)r{rrBrrrsz*VOCSegDataset.__init__..r examples)rr Normalizer crop_sizerfilterrrrrrrVr6)rCrnrrrrrrBrrDs  zVOCSegDataset.__init__cCs||dS)N)rr)rCrrrrrszVOCSegDataset.normalize_imagecsfdd|DS)Ncs8g|]0}|jdjdkr|jdjdkr|qS)r-rr_)rXr)r{rrBrrrsz(VOCSegDataset.filter..r)rCrrrBrrszVOCSegDataset.filtercCs2t|j||j|f|j\}}|t||jfSr)rrrrrr)rCrrrCrrrrs zVOCSegDataset.__getitem__cCs t|jSrrrBrrrr.szVOCSegDataset.__len__N) rPrQrRrSrDrrrr.rrrrrs  rcCsdtdtjdd}t}tjjj t d|||dd|d}tjjj t d|||d|d}||fS)z_Load the VOC semantic segmentation dataset. Defined in :numref:`sec_semantic_segmentation`r VOCdevkitVOC2012T)rk drop_lastrF)rr) rr&rrrrrbrrrlr)rgrrrrrrrr load_data_vocs&   rzkaggle_cifar10_tiny.zip(2068874e4b9a9f0fb07ebe0ad2b29754449ccacd cifar10_tinyc CsFt|d}|dd}W5QRXdd|D}tdd|DS)zcRead `fname` to return a filename to label dictionary. Defined in :numref:`sec_kaggle_cifar10`rr-NcSsg|]}|dqS),)rstriprr rrrrsz#read_csv_labels..css|]\}}||fVqdSrr)r{rrCrrrrsz"read_csv_labels..)r r{r)rrr|rrrrread_csv_labelss rcCstj|ddt||dS)zQCopy a file into a target directory. Defined in :numref:`sec_kaggle_cifar10`TrN)rrshutilcopy)filename target_dirrrrcopyfilesrc Cst|dd}tdt||}i}ttj |dD]}|| dd}tj |d|}t |tj |dd|||ks|||krt |tj |dd|| |dd||<qFt |tj |dd|qF|S) zgSplit the validation set out of the original training set. Defined in :numref:`sec_kaggle_cifar10`rGr-rrrtrain_valid_test train_validvalid)rrr most_commonr;rfloorrlistdirrrrrr) r#r valid_ratiorn_valid_per_label label_count train_filerCrrrrreorg_train_valids*rc CsBttj|dD](}ttj|d|tj|dddqdS)ziOrganize the testing set for data loading during prediction. Defined in :numref:`sec_kaggle_cifar10`testrunknownN)rrrrr)r# test_filerrr reorg_tests  rzkaggle_dog_tiny.zip(0cb91d09b814ecdc07b50f31f8dcad3e81d6a86ddog_tinyzptb.zip(319d85e578af0cdc590547f26231e4e31cdf1e42ptbc CsDtd}ttj|d}|}W5QRXdd|dDS)z[Load the PTB dataset into a list of text lines. Defined in :numref:`sec_word2vec_data`rz ptb.train.txtcSsg|] }|qSrrryrrrrszread_ptb..r)rr&r rrrr r)r#rraw_textrrrread_ptbs rcsLfdd|D}t|tfddfdd|DfS)zKSubsample high-frequency words. Defined in :numref:`sec_word2vec_data`csg|]}fdd|DqS)csg|]}|jkr|qSr)rrrrrrs(subsample...rryrrrrszsubsample..cs"tddtd|kS)Nrr-g-C6?)runiformrrr)rrrrrus zsubsample..keepcsg|]}fdd|DqS)csg|]}|r|qSrrrrurrr!srrryrrrr!s)rrrJr) sentencesrr)rrurrr subsamples   rc sgg}}|D]tdkr q|7}ttD]\}td|}tttd||tt|d|}|||fdd|Dq4q||fS)z_Return center words and context words in skip-gram. Defined in :numref:`sec_word2vec_data`r_r-rcsg|] }|qSrr)r{rrzrrr5sz,get_centers_and_contexts..) r6rrrr1r;r[removerH)rmax_window_sizecenterscontextsr| window_sizerrrrget_centers_and_contexts$s    rc@s eZdZdZddZddZdS)RandomGeneratorz@Randomly draw among {1, ..., n} according to n sampling weights.cCs.ttdt|d|_||_g|_d|_dS)z&Defined in :numref:`sec_word2vec_data`r-rN)r1rr6 populationsampling_weights candidatesr|)rCrrrrrD:szRandomGenerator.__init__cCsJ|jt|jkr,tj|j|jdd|_d|_|jd7_|j|jdS)Nr)r_rr-)r|r6rrchoicesrrrBrrrdrawBszRandomGenerator.drawN)rPrQrRrSrDrrrrrr8src sxfddtdtD}gt|}}|D]B}g}t|t||krh|} | |kr8|| q8||q0|S)zTReturn noise words in negative sampling. Defined in :numref:`sec_word2vec_data`csg|]}|dqS)g?)rrzrrrrrQsz!get_negatives..r-)rr6rrrH) all_contextsrrr6r all_negatives generatorr negativesnegrrr get_negativesKs    r c Cstdd|D}ggggf\}}}}|D]\}}}t|t|} ||g7}|||dg|| g7}|dg| dg|| g7}|dgt|dg|t|g7}q*tt|dt|t|t|fS)zpReturn a minibatch of examples for skip-gram with negative sampling. Defined in :numref:`sec_word2vec_data`css$|]\}}}t|t|VqdSrr )r{rr.rrrrrbszbatchify..rr-rU)r;r6rrYr) rrrcontexts_negativesmasksrr/contextnegativecur_lenrrrbatchify^s (rcst}t}tj|ddt|\}}fdd|D}t||\}} t| ||} Gdddtjj j } | || | } tjj j | |dt |d} | fS) zrDownload the PTB dataset and then load it into memory. Defined in :numref:`subsec_word2vec-minibatch-loading`rrcsg|] }|qSrrryrrrrusz!load_data_ptb..c@s$eZdZddZddZddZdS)z!load_data_ptb..PTBDatasetcSs<t|t|kr t|ks&nt||_||_||_dSr)r6rrrr )rCrrr rrrrD|s&z*load_data_ptb..PTBDataset.__init__cSs|j||j||j|fSr)rrr )rCrrrrrsz-load_data_ptb..PTBDataset.__getitem__cSs t|jSr)r6rrBrrrr.sz)load_data_ptb..PTBDataset.__len__N)rPrQrRrDrr.rrrr PTBDataset{srT)rkZ collate_fnr) rrrrrrr rbrrDatasetrlr)rgrnum_noise_wordsrr subsampledrr all_centersrr rrorrrr load_data_ptbms, rzglove.6B.50d.zip(0b8703943ccdb6eb788e6f091b8946e82231bc4dz glove.6b.50dzglove.6B.100d.zip(cd43bfb07e44e6f27cbcc7bc9ae3d80284fdaf5az glove.6b.100dzglove.42B.300d.zip(b5116e234e9eb9076672cfeabf5469f3eec904fazglove.42b.300dz wiki.en.zip(c1816da3821ae9f43899be655002f6c723e91b88zwiki.enc@s0eZdZdZddZddZddZdd Zd S) TokenEmbeddingzToken Embedding.cCs2||\|_|_d|_ddt|jD|_dS)z!Defined in :numref:`sec_synonyms`rcSsi|]\}}||qSrrrrrrrsz+TokenEmbedding.__init__..N)_load_embeddingr idx_to_vec unknown_idxrr)rCembedding_namerrrrDs zTokenEmbedding.__init__c Csdgg}}t|}ttj|dd^}|D]R}|d}|ddd|ddD}}t|dkr0| || |q0W5QRXdgt|dg|}|t |fS) Nrzvec.txtrrtrcSsg|] }t|qSrr)r{elemrrrrsz2TokenEmbedding._load_embedding..r-) rr&r rrrrrr6rHr) rCr#rr!r#rrzelemsrrrrr s     zTokenEmbedding._load_embeddingcs&fdd|D}jt|}|S)Ncsg|]}j|jqSr)rrr"rrBrrrsz.TokenEmbedding.__getitem__..)r!rr)rCrrvecsrrBrrs  zTokenEmbedding.__getitem__cCs t|jSrrrBrrrr.szTokenEmbedding.__len__N)rPrQrRrSrDr rr.rrrrrs rcCsVdg|dg}dgt|d}|dk rN||dg7}|dgt|d7}||fS)z_Get tokens of the BERT input sequence and their segment IDs. Defined in :numref:`sec_bert`rr_Nr-r )tokens_atokens_brsegmentsrrrget_tokens_and_segmentss r,cs*eZdZdZdfdd ZddZZS) BERTEncoderz=BERT encoder. Defined in :numref:`subsec_bert_input_rep`rc  stt|jf| t|||_td||_t|_t |D],}|j |t | | | ||||||d q@t td| ||_dS)Nr_Tr-)r`r-rDrr4token_embeddingsegment_embeddingrrrrrr Parameterrbrandn pos_embedding)rCrrrrrrrr3rrrrrr|rjrrrDs(   zBERTEncoder.__init__cCsV||||}||jjddd|jdddf}|jD]}|||}qB|Sr)r/r0r3rrXr)rCrr+rwr3rrrrros (  zBERTEncoder.forward)rr.r.r.rprrrjrr-sr-cs*eZdZdZdfdd ZddZZS)MaskLMzWThe masked language model task of BERT. Defined in :numref:`subsec_bert_input_rep`r.c sBtt|jf|tt||tt|t|||_dSr) r`r4rDrrrCrrmlp)rCrr num_inputsrrjrrrDs  zMaskLM.__init__cCs`|jd}|d}|jd}td|}t||}|||f}|||df}||}|S)Nr-rGr)rXrYrbr:rvr5)rCr3pred_positionsnum_pred_positionsrg batch_idxmasked_X mlm_Y_hatrrrros       zMaskLM.forward)r.rprrrjrr4sr4cs(eZdZdZfddZddZZS)NextSentencePredzOThe next sentence prediction task of BERT. Defined in :numref:`subsec_mlm`c s$tt|jf|t|d|_dS)Nr_)r`r<rDrrCr)rCr6rrjrrrD szNextSentencePred.__init__cCs ||Sr)rrrrrro szNextSentencePred.forwardrprrrjrr< s r<cs,eZdZdZd fdd Zd ddZZS) BERTModelz4The BERT model. Defined in :numref:`subsec_nsp`rr.csftt|t||||||||| | | | d |_tt| |t|_ t ||||_ t ||_ dS)N)rrrr)r`r=rDr-r-rrrCTanhhiddenr4mlmr<nsp)rCrrrrrrrr3rrrrZhid_in_featuresZmlm_in_featuresZnsp_in_featuresrjrrrD s"zBERTModel.__init__Nc CsT||||}|dk r$|||}nd}|||dddddf}|||fSr)r-r@rAr?)rCrr+rwr7 encoded_Xr; nsp_Y_hatrrrro s "zBERTModel.forward)rr.r.r.r.r.r.)NNrprrrjrr= sr=)zHhttps://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip(3c914d17d80b1459be871a5039ac23e752a53cbe wikitext-2c CsHtj|d}t|d}|}W5QRXdd|D}t||S)%Defined in :numref:`sec_bert-dataset`zwiki.train.tokensrcSs0g|](}t|ddkr|dqS)z . r_)r6rrwrxryrrrr5 sz_read_wiki..)rrrr r{rrk)r# file_namerr| paragraphsrrr _read_wiki/ s  rIcCs0tdkrd}ntt|}d}|||fS)rFrTF)rchoice)sentence next_sentencerHis_nextrrr_get_next_sentence: s  rNc Csvg}tt|dD]\}t||||d|\}}}t|t|d|krPqt||\} } || | |fq|S)rFr-r)rr6rNrr,rH) paragraphrHrrnsp_data_from_paragraphr|r)r*rMrr+rrr_get_nsp_data_from_paragraphD s  rQcCsdd|D}g}t||D]f}t||kr4qd}tdkrJd}n"tdkr`||}n t|j}|||<||||fq ||fS)rFcSsg|]}|qSrrrrrrrV sz'_replace_mlm_tokens..Ng?r)rrkr6rJrrH)rcandidate_pred_positions num_mlm_predsrmlm_input_tokenspred_positions_and_labelsmlm_pred_position masked_tokenrrr_replace_mlm_tokensQ s"       rYc Csg}t|D]\}}|dkrq ||q tdtt|d}t||||\}}t|ddd}dd|D}d d|D} ||||| fS) ,Defined in :numref:`subsec_prepare_mlm_data`)r'r(r-333333?cSs|dSrrrrrrr} rz+_get_mlm_data_from_tokens..)rcSsg|] }|dqSrrr{vrrrr~ sz-_get_mlm_data_from_tokens..cSsg|] }|dqSrrr\rrrr s)rrHr;roundr6rYr) rrrSr|rrTrUrVr7mlm_pred_labelsrrr_get_mlm_data_from_tokensn s$ r`c Cs^t|d}ggg}}}ggg}}} g} |D]\} } } }}|tj| |dg|t| tjd|tj|dg|t|tjd|tjt| tjd|tj| dg|t| tjd|tjdgt| dg|t| tjd| tj| dg|t| tjd| tj|tjdq4|||||| | fS)rZr[rrrrr)r^rHrbrr6rr)examplesrrmax_num_mlm_preds all_token_ids all_segmentsrwall_pred_positionsall_mlm_weightsall_mlm_labels nsp_labels token_idsr7mlm_pred_label_idsr+rMrrr_pad_bert_inputs sR      rkc@s(eZdZdZddZddZddZdS) _WikiTextDatasetrZcsdd|D}dd|D}tj|dddddgd _g}|D]}|t||j|q>fd d|D}t||j\____ _ _ _ dS) NcSsg|]}tj|ddqS)r~r)rr)r{rOrrrr s z-_WikiTextDataset.__init__..cSsg|]}|D]}|q qSrr)r{rOrKrrrr srLrrRr'r(rcs&g|]\}}}t|j||fqSr)r`r)r{rr+rMrBrrr s ) rrrextendrQrkrcrdrwrerfrgrh)rCrHrrrarOrrBrrD s@  z_WikiTextDataset.__init__cCs<|j||j||j||j||j||j||j|fSr)rcrdrwrerfrgrhrrrrr sz_WikiTextDataset.__getitem__cCs t|jSr)r6rcrBrrrr. sz_WikiTextDataset.__len__Nrrrrrrl srlcCsFt}tdd}t|}t||}tjjj||d|d}||j fS)zNLoad the WikiText-2 dataset. Defined in :numref:`subsec_prepare_mlm_data`rETr) rrr&rIrlrbrrrlr)rgrrr#rH train_setrrrrload_data_wiki s   roc Csn||||d|\} } } || d||d|dd} | |d} || | }| |}| ||fS)z)Defined in :numref:`sec_bert-pretraining`rGr-g:0yE>)rYrJ)rrrZtokens_XZ segments_XZ valid_lens_xZpred_positions_XZ mlm_weights_XZmlm_YZnsp_yrr;rCmlm_lnsp_lrrrr_get_batch_loss_bert s   rr)z>http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz(01ada507287d82875905620988597833ad4e0903aclImdbc Csgg}}dD]}tj||r"dnd|}t|D]Z}ttj||d<}|ddd}||||dkr~d nd W5QRXq4q||fS) z_Read the IMDb review dataset text sequences and labels. Defined in :numref:`sec_sentiment`)rr rrrzutf-8rrrr-r) rrrrr r decoderrH) r#rnrrrC folder_namefilerreviewrrr read_imdb s  $ryc stdd}t|d}t|d}tj|ddd}tj|ddd}tj|ddtfd d |D}tfd d |D}t|t|d f|} tj|t|d f|dd } | | fS)zlReturn data iterators and the vocabulary of the IMDb review dataset. Defined in :numref:`sec_sentiment`rtTFrr~rrLrcs"g|]}t|dqSrrrryrrrr s z"load_data_imdb..cs"g|]}t|dqSrr{ryrrrr s r-r)rr&ryrrrbrrp) rgrr# train_data test_data train_tokens test_tokenstrain_features test_featuresrrrrrload_data_imdb s(   rcCsDtj||td}tj||dddd}|dkr@dSdS)zUPredict the sentiment of a text sequence. Defined in :numref:`sec_sentiment_rnn`r=r-rGrpositiver)rbrrrr1rrY)rrsequencerCrrrpredict_sentiment sr)z3https://nlp.stanford.edu/projects/snli/snli_1.0.zip(9fcde07509c7e87ec61c640c1b2753d9041758e4SNLIc sddddddtj||r$dnd}t|d  }d d |dd D}W5QRXfd d |D}fdd |D}fdd |D}|||fS)zRead the SNLI dataset into premises, hypotheses, and labels. Defined in :numref:`sec_natural-language-inference-and-dataset`cSs2tdd|}tdd|}tdd|}|S)Nz\(rz\)z\s{2,}rt)rurvrw)rrrr extract_text szread_snli..extract_textrr-r_) entailment contradictionneutralzsnli_1.0_train.txtzsnli_1.0_test.txtrcSsg|]}|dqS)rrr{rowrrrr# szread_snli..Ncs$g|]}|dkr|dqS)rr-rrr label_setrrr$ s cs$g|]}|dkr|dqS)rr_rrrrrr% s cs$g|]}|dkr|dqSrrr)rrrr& s )rrrr r{)r#rnrGrrowspremises hypothesesrrrr read_snli s  $rc@s2eZdZdZd ddZddZddZd d ZdS) SNLIDatasetzsA customized dataset to load the SNLI dataset. Defined in :numref:`sec_natural-language-inference-and-dataset`NcCs||_t|d}t|d}|dkrDtj||ddgd|_n||_|||_|||_t |d|_ t dt t |jddS) Nrr-rLrrr_rr)rrrrr_padrrrbrrrrVr6)rCrorrall_premise_tokensall_hypothesis_tokensrrrrD- s    zSNLIDataset.__init__cstfdd|DS)Ncs(g|] }tj|jjdqSr)rrrrryrBrrr< s z$SNLIDataset._pad..)rbr)rCr|rrBrr; szSNLIDataset._padcCs|j||j|f|j|fSr)rrrrrrrr@ szSNLIDataset.__getitem__cCs t|jSr)r6rrBrrrr.C szSNLIDataset.__len__)N)rPrQrRrSrDrrr.rrrrr) s  rrc Csvt}td}t|d}t|d}t||}t|||j}tjjj ||d|d}tjjj ||d|d} || |jfS)zDownload the SNLI dataset and return data iterators and vocabulary. Defined in :numref:`sec_natural-language-inference-and-dataset`rTFr) rrr&rrrrbrrrl) rgrrr#r|r}rntest_setrrrrrload_data_snliF s      rcCsr|tj||td}tj||td}tj||d|dgdd}|dkrbdS|dkrndSdS) zPredict the logical relationship between the premise and hypothesis. Defined in :numref:`sec_natural-language-inference-attention`r=)r-rGr-rrrrr)rrbrrr1rrY)rrpremise hypothesisrCrrr predict_snliX src Cs|jd}tj|f|jd}tj|f|jd}|||} ||} || } || || j|| || jd} | | | S)z=Update discriminator. Defined in :numref:`sec_basic_gan`rr=r_) rXrbonesr0r5rdetachrYrr) r3Znet_Dnet_Gr trainer_Drgrr5real_Yfake_Xfake_Yloss_Drrrupdate_Dd s  rc CsZ|jd}tj|f|jd}|||}||}||||j} | || S)z9Update generator. Defined in :numref:`sec_basic_gan`rr=)rXrbrr0rrYrr) rrrr trainer_Grgrrrloss_Grrrupdate_Gw s rz pokemon.zip(c065c0e2593b8b161a2d7873e42418bf6a21106cpokemoncOs|j||Sr)rrr<rrrrrr rrcOs |j||Sr)rrrrrr rcOs |j||Sr)rYrrrrr rcOs |j||Srr:rrrrr rcOs |j||Sr)rJrrrrr rcOs |j||Sr)rrrrrr rcOs |j||Sr)rBrrrrr rcOs |j||Sr)trrrrr r)r) NNNNNNr)r)r*rN)T)Nr)N)r)N)r)N)r~)rG)Fr)F)N)r)r)F)Nrhri)rN)rr)r_)r)r-)NN)r)rZ)rry)T)T)N)rz)r)rrDATA_URLrrLrbrPILrrZtorch.nnrrlZ torch.utilsrrr nn_Modulerr rrrrursysrrErrpandasrrIPythonr matplotlibr rmatplotlib_inliner modulesrPrrrr(r9r?r]r^rarirprrrrrrrrrrrrrrrr&r'r1r4r9rAr[r\r}rrrrrrrrrrrrrrrrr rrrr%r&r*r,r2r>rUr?rOrZrgrtrxryrrrrrrrrrrrrrrrrrrrrrr r rr)r:rGrYrbrkrrrxrrrrrrrr VOC_CLASSESrrrrrrrrrrrrrr rrrr,r-r4r<r=rIrNrQrYr`rkrlrorrryrrrrrrrrrr5rr:rrsinhrcoshrlinspacerr[rVrandrWrrrrHrabseyerrYr;rrrrrrrrsv                 !      )     *  ,   & +      ) $  )              '   $  #    (                    #        #     $