U 0Jd2d3Z?Gd4d5d5Z@d6d7ZAdd9d:ZBd;d<ZCeZdZejDEd=d>fd?d@ZFddAdBZGdCdDZHedEdFfedG<edHdIfedJ<ddKdLZIdMdNZJdOdPZKddQdRZLdSdTZMGdUdVdVejZNe-jdWdXfe-jdY<dZd[ZOdd]d^ZPGd_d`d`ZQdadbZRddddeZSdfdgZTdhdiZUGdjdkdkZVddmdnZWGdodpdpZXdqdrZYdsdtZZdudvZ[ddwdxZ\GdydzdzejZ]e-jd{d|fe-jd}<d~dZ^ddZ_dddZ`ddZaddZbddZcdddZdGdddejZeGdddejZfGdddejZgGddde-jeZhGddde jijjZkddZldddZmddZndddZoddZpGdddejZqGdddejZrGddde-jfZsGdddejZtddZuddZvGdddejZwGdddejZxGdddejZyGdddejZzGddde-jeZ{ddZ|dddZ}ddZ~e-jddfe-jd<dddƄZdddɄZddd˄ZGdd̈́d̓ZddτZddфZe-jfddӄZe-jfddՄZe-Je-jfddׄZe-jddfe-jd<dd܄ZddބZddZddZdddZddZdddZdddZddZddZddZdddZe-jddfe-jd<dddZGddde jjZddZe-jddfe-jd<dҐddZdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddggZddd d d d d ddddddddddddddgZddZddZd d!ZGd"d#d#e jjZd$d%Ze-jd&d'fe-jd(<d)d*Zd+d,Zd-d.Zd/d0Ze-jd1d2fe-jd3<e-jd4d5fe-jd6<d7d8Zd9d:Zd;d<ZGd=d>d>Zd?d@ZdAdBZdCdDZe-jdEdFfe-jdG<e-jdHdIfe-jdJ<e-jdKdLfe-jdM<e-jdNdOfe-jdP<GdQdRdRZdӐdSdTZGdUdVdVejZGdWdXdXejZGdYdZdZejZGd[d\d\ejZd]e-jd^<d_d`ZdadbZdcddZdedfZdgdhZdidjZGdkdldle jjZdmdnZdodpZdqe-jdr<dsdtZdԐdvdwZdxdyZdze-jd{<d|d}ZGd~dde jjZdՐddZddZddZde-jd<ddZÐd֐ddZĐdאddZŐdؐddZe-JdfddZGddde jijȃZGddde jijȃZʐddZːddZ̐dِddZe-jddfe-jd<Gddde jjZΐddZϐddZe-jddfe-jd<ddZѐddZe jZe jZe jZe jZe jZe jZe jZe jZe jZe jZe jZe jZe jZe jjZe jjZe jZe jZe jZe jZe jZe jZe jZddZddZddZddZddZddZdS(TFz,http://d2l-data.s3-accelerate.amazonaws.com/)autogradcontextgluonimageinitnpnpx)nnrnn) transformsN) defaultdict)display)pyplot)backend_inlinecCstddS)zWUse the svg format to display a plot in Jupyter. Defined in :numref:`sec_calculus`svgN)rZset_matplotlib_formatsrr8/home/d2l-worker/workspace/d2l-en-release/./d2l/mxnet.pyuse_svg_display*srg @@cCst|tjjd<dS)zJSet the figure size for matplotlib. Defined in :numref:`sec_calculus`zfigure.figsizeN)rd2lpltZrcParamsfigsizerrr set_figsize0srcCsV|||||||||||||rJ|||dS)zCSet the axes for matplotlib. Defined in :numref:`sec_calculus`N) set_xlabel set_ylabelZ set_xscaleZ set_yscaleZset_xlimZset_ylimlegendgrid)axesxlabelylabelxlimylimxscaleyscalerrrrset_axes7s       r&linear-zm--zg-.zr:c  Cs|dkr g}t| | r| ntj} dd} | |r<|g}|dkrZggt||}}n| |rh|g}t|t|kr|t|}| t||| D].\} }}t| r| | ||q| ||qt| |||||||dS)z8Plot data points. Defined in :numref:`sec_calculus`NcSs.t|dr|jdkp,t|to,t|dd S)Nndimr__len__)hasattrr* isinstancelistXrrr has_one_axisRszplot..has_one_axis) rrrgcalenclazipplotr&)r1Yr r!rr"r#r$r%fmtsrrr2xyfmtrrrr7Es& r7c@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)TimerzRecord multiple running times.cCsg|_|dS)z(Defined in :numref:`subsec_linear_model`N)timesstartselfrrr__init__hszTimer.__init__cCst|_dS)zStart the timer.N)timetikr@rrrr?msz Timer.startcCs |jt|j|jdS)z-Stop the timer and record the time in a list.)r>appendrCrDr@rrrstopqsz Timer.stopcCst|jt|jS)zReturn the average time.)sumr>r4r@rrravgvsz Timer.avgcCs t|jS)zReturn the sum of time.)rHr>r@rrrrHzsz Timer.sumcCst|jS)zReturn the accumulated time.)rarrayr>cumsumtolistr@rrrrK~sz Timer.cumsumN) __name__ __module__ __qualname____doc__rBr?rGrIrHrKrrrrr=fsr=cCsJtdd|t|f}t|||}|tdd|j7}|t|dfS)zIGenerate y = Xw + b + noise. Defined in :numref:`sec_linear_scratch`rr+{Gz?rEr+)rnormalr4matmulshapereshape)wb num_examplesr1r;rrrsynthetic_datasrZcCst|||S)zIThe linear regression model. Defined in :numref:`sec_linear_scratch`)rrT)r1rWrXrrrlinregsr[cCs|t||jddS)z:Squared loss. Defined in :numref:`sec_linear_scratch`)rrVrU)y_hatr;rrr squared_losssr^cCs(|D]}|||j||dd<qdS)zSMinibatch stochastic gradient descent. Defined in :numref:`sec_linear_scratch`N)grad)paramslr batch_sizeparamrrrsgdsrdcCstjj|}tjj|||dS)zMConstruct a Gluon data iterator. Defined in :numref:`sec_linear_concise`shuffle)rdata ArrayDataset DataLoader) data_arraysrbis_traindatasetrrr load_arrays rmc s*ddddddddd d g fd d |DS) z]Return text labels for the Fashion-MNIST dataset. Defined in :numref:`sec_fashion_mnist`zt-shirtZtrouserZpulloverZdressZcoatZsandalZshirtZsneakerZbagz ankle bootcsg|]}t|qSr)int.0iZ text_labelsrr sz,get_fashion_mnist_labels..r)labelsrrrrget_fashion_mnist_labelss ru?c Cs||||f}tjj|||d\}}|}tt||D]N\}\} } | t| | j d| j d|r<| ||q<|S)zBPlot a list of images. Defined in :numref:`sec_fashion_mnist`rF) rrsubplotsflatten enumerater6imshownumpyrZ get_xaxisZ set_visibleZ get_yaxis set_title) imgsnum_rowsnum_colstitlesscaler_rrqaximgrrr show_imagessrcCstjdrdSdS)z`Use 4 processes to read the data except for Windows. Defined in :numref:`sec_fashion_mnist`winr)sysplatform startswithrrrrget_dataloader_workerssrcCstjj}|jg}|r,|d|j||j|}|jdd |}|jdd |}tjj ||dt dtjj ||dt dfS)zlDownload the Fashion-MNIST dataset and then load it into memory. Defined in :numref:`sec_fashion_mnist`rT)trainFrf num_workers) rrgZvisionr ZToTensorinsertZResizeZComposeZ FashionMNISTZtransform_firstrir)rbresizerltransZ mnist_trainZ mnist_testrrrload_data_fashion_mnists    rcCsTt|jdkr*|jddkr*tj|dd}t||j|k}ttt||jS)zXCompute the number of correct predictions. Defined in :numref:`sec_softmax_scratch`r+axis)r4rUrargmaxastypedtypefloat reduce_sum)r]r;cmprrraccuracysrcCsBtd}|D]$\}}|t|||t|q |d|dS)z\Compute the accuracy for a model on a dataset. Defined in :numref:`sec_softmax_scratch`r\rr+) Accumulatoraddrrsize)net data_itermetricr1r;rrrevaluate_accuracys rc@s0eZdZdZddZddZddZdd Zd S) rz)For accumulating sums over `n` variables.cCsdg||_dS)(Defined in :numref:`sec_softmax_scratch`Nrg)rAnrrrrBszAccumulator.__init__cGsddt|j|D|_dS)NcSsg|]\}}|t|qSrr)rparXrrrrssz#Accumulator.add..)r6rgrAargsrrrrszAccumulator.addcCsdgt|j|_dS)Nr)r4rgr@rrrresetszAccumulator.resetcCs |j|SNrrAidxrrr __getitem__szAccumulator.__getitem__N)rMrNrOrPrBrrrrrrrrs rc Cstd}t|tjr|j}|D]b\}}t||}|||}W5QRX|||jd| t | t |||j q|d|d|d|dfS)zdTrain a model within one epoch (defined in Chapter 3). Defined in :numref:`sec_softmax_scratch`rr\r+)rr.rTrainersteprrecordbackwardrUrrrHrr) r train_iterlossupdaterrr1r;r]lrrrtrain_epoch_ch3s    rc @s"eZdZdZd ddZd d ZdS) AnimatorzFor plotting data in animation.Nr'r(r+rc szdkr gttjj| | | d\__| | dkrDjg_fdd_dd|___ dS)rNrr+c stjdSNr)rr&rrrrAr r"r$r!r#r%rrsz#Animator.__init__..) rrrrwfigr config_axesr1r8r9) rAr r!rr"r#r$r%r9nrowsncolsrrrrrBs  zAnimator.__init__cCst|ds|g}t|}t|ds,|g|}|jsFddt|D|_|js`ddt|D|_tt||D]<\}\}}|dk rn|dk rn|j|||j||qn|jd t|j|j|j D]\}}}|jd |||q| t |jt jdddS)Nr,cSsg|]}gqSrrrprrrrrs!sz Animator.add..cSsg|]}gqSrrrrrrrs#srT)wait)r-r4r1ranger8ryr6rFrr5r9r7rr rZ clear_output)rAr:r;rrqrrXr<rrrrs&    z Animator.add) NNNNNr'r'r(r+r+r)rMrNrOrPrBrrrrrrs rc Cstdd|gddgdddgd}t|D]2}t||||}t||} ||d|| fq&|\} } | d ksrt| | dkr| d kst| | dkr| d kst| d S) zSTrain a model (defined in Chapter 3). Defined in :numref:`sec_softmax_scratch`epochr+g333333?g? train loss train acctest accr r"r#r?gffffff?N)rrrrrAssertionError) rr test_iterr num_epochsranimatorrZ train_metricstest_accZ train_loss train_accrrr train_ch3/s  rcCs||D] \}}qqt|}ttj||dd}ddt||D}tjt|d||ddfd||d|ddS) zTPredict labels (defined in Chapter 3). Defined in :numref:`sec_softmax_scratch`r+rcSsg|]\}}|d|qS) r)rptruepredrrrrsFszpredict_ch3..r)rN)rrurr6rrV)rrrr1r;Ztruespredsrrrr predict_ch3>s   rcCsNtd}|D].\}}||||}|t|t|q|d|dS)z`Evaluate the loss of a model on the given dataset. Defined in :numref:`sec_model_selection`r\rr+)rrrrr)rrrrr1r;rrrr evaluate_lossJs   rz..rgc Cs|tkst|dtdt|\}}tj|ddtj||dd}tj|rt }t |d"}| d}|s~q| |qnW5QRX| |kr|Std |d |d tj|ddd }t |d }||jW5QRX|S)zmDownload a file inserted into DATA_HUB, return the local filename. Defined in :numref:`sec_kaggle_house`z does not exist in .Texist_ok/rErbiz Downloading z from z...)streamverifywb)DATA_HUBrosmakedirspathjoinsplitexistshashlibsha1openreadupdate hexdigestprintrequestsgetwritecontent) nameZ cache_dirurlZ sha1_hashfnamerfrgrrrrdownloadWs$      rcCs|t|}tj|}tj|\}}|dkr:t|d}n"|dkrPt|d}n ds\t d| ||rxtj ||S|S)zODownload and extract a zip/tar file. Defined in :numref:`sec_kaggle_house`z.zipr)z.tarz.gzFz$Only zip/tar files can be extracted.) rrrdirnamesplitextzipfileZipFiletarfilerr extractallr)rZfolderrbase_dirdata_dirextfprrrdownload_extractos   rcCstD] }t|qdS)zNDownload all files in the DATA_HUB. Defined in :numref:`sec_kaggle_house`N)rr)rrrr download_allsrzkaggle_house_pred_train.csvZ(585e9cc93e70b39160e7921475f9bcd7d31219ceZkaggle_house_trainzkaggle_house_pred_test.csvZ(fa19780a7b011d9b009e8bff8e99922a8ee2eb90Zkaggle_house_testcCs"t|dkrt|StS)zVReturn gpu(i) if exists, otherwise return cpu(). Defined in :numref:`sec_use_gpu`r+)rnum_gpusgpucpu)rqrrrtry_gpusr cCs(ddttD}|r|StgS)z]Return all available GPUs, or [cpu()] if no GPU exists. Defined in :numref:`sec_use_gpu`cSsg|]}t|qSr)rrrorrrrssz try_all_gpus..)rrrrdevicesrrr try_all_gpussr c Cs|j\}}t|jd|d|jd|df}t|jdD]F}t|jdD]2}t|||||||f||||f<qTqB|S)zFCompute 2D cross-correlation. Defined in :numref:`sec_conv_layer`rr+)rUrzerosrr)r1KhrWr8rqjrrrcorr2ds  *2rcCs||s t|dd}td}|D]<\}}||||}}|t|||t |q.|d|dS)z^Compute the accuracy for a model on a dataset using a GPU. Defined in :numref:`sec_lenet`rr\r+) r/collect_paramsvalueslist_ctxrr as_in_ctxrrr)rrdevicerr1r;rrrevaluate_accuracy_gpus   rc Cs|jd|tdtj}t|dd|i}tj dd|gddd gd }t t |} } t |D](} t d } t|D]\} \}}| ||||}}t||}|||}W5QRX|||jd | |t|||jd | | d | d }| d| d }| d| dd ksR| | dkr|| | d| ||dfqt||}|| ddd|fqhtd|dd|dd|dt| d || ddt|dS)zTTrain a model with a GPU (defined in Chapter 6). Defined in :numref:`sec_lenet`T) force_reinitctxrrd learning_raterr+rrr)r r"rrrr\Nloss .3f , train acc , test acc .1f examples/sec on ) initializerXavierrrSoftmaxCrossEntropyLossrrrrr=r4rrryr?rrrrrrUrrHrrGrrstr)rrrrrarrtrainerrtimer num_batchesrrrqr1r;r]rtrain_lrrrrr train_ch6s@      "  r*cs*eZdZdZdfdd ZddZZS) ResidualzThe Residual block of ResNet.Fr+c sjtjf|tj|dd|d|_tj|ddd|_|rLtj|d|d|_nd|_t|_t|_ dS)Nrr+) kernel_sizepaddingstrides)r,r-)r,r.) superrBr Conv2Dconv1conv2conv3 BatchNormbn1bn2)rA num_channels use_1x1convr.kwargs __class__rrrBs   zResidual.__init__cCsDt|||}|||}|jr6||}t||Sr)rrelur5r1r6r2r3rAr1r8rrrforwards  zResidual.forward)Fr+rMrNrOrPrBr> __classcell__rrr:rr+s r+ztimemachine.txtZ(090b5e7e70c295757f55df93cb0a180b9691891a time_machinec Cs2ttdd}|}W5QRXdd|DS)ziLoad the time machine dataset into a list of text lines. Defined in :numref:`sec_text_preprocessing`rArcSs"g|]}tdd|qS)z [^A-Za-z]+ )resubstriplowerrplinerrrrssz%read_time_machine..)rrr readlines)rlinesrrrread_time_machinesrKwordcCs<|dkrdd|DS|dkr,dd|DStd|dS)z`Split text lines into word or character tokens. Defined in :numref:`sec_text_preprocessing`rLcSsg|] }|qSrrrGrrrrssztokenize..charcSsg|] }t|qSr)r/rGrrrrsszERROR: unknown token type: N)r)rJtokenrrrtokenizes rPc@sJeZdZdZdddZddZdd Zd d Zed d Z eddZ dS)VocabzVocabulary for text.NrcCs|dkr g}|dkrg}t|}t|dddd|_dg||_ddt|jD|_|jD]>\}}||krtq||jkr`|j|t|jd |j|<q`dS) z+Defined in :numref:`sec_text_preprocessing`NcSs|dSNr+rr:rrrrz Vocab.__init__..TkeyreversecSsi|]\}}||qSrrrprrOrrr sz"Vocab.__init__..r+) count_corpussorteditems _token_freqs idx_to_tokenry token_to_idxrFr4)rAtokensmin_freqreserved_tokenscounterrOfreqrrrrBs$   zVocab.__init__cCs t|jSrr4r_r@rrrr,sz Vocab.__len__cs0t|ttfsj|jSfdd|DS)Ncsg|]}|qSr)rrprOr@rrrs!sz%Vocab.__getitem__..)r.r/tupler`runk)rArarr@rrszVocab.__getitem__cs*t|ttfsj|Sfdd|DS)Ncsg|]}j|qSrr_)rpindexr@rrrs&sz#Vocab.to_tokens..)r.r/rhr_)rAindicesrr@r to_tokens#s zVocab.to_tokenscCsdSrrr@rrrri(sz Vocab.unkcCs|jSr)r^r@rrr token_freqs,szVocab.token_freqs)NrN) rMrNrOrPrBr,rrmpropertyrirnrrrrrQs  rQcCs2t|dkst|dtr(dd|D}t|S)zICount token frequencies. Defined in :numref:`sec_text_preprocessing`rcSsg|]}|D]}|q qSrrrprHrOrrrrs7sz count_corpus..)r4r.r/ collectionsCounter)rarrrr[0sr[rEcsFt}t|d}t|fdd|D}|dkr>|d|}|fS)zuReturn token indices and the vocabulary of the time machine dataset. Defined in :numref:`sec_text_preprocessing`rNcsg|]}|D] }|q qSrrrpvocabrrrsCsz,load_corpus_time_machine..rN)rKrPrQ) max_tokensrJracorpusrrsrload_corpus_time_machine:s  rwc #stdddtd}ttd|}t|fdd||}td|||D]N}||||}fdd|D}fdd|D} t|t| fVqldS) zhGenerate a minibatch of subsequences using random sampling. Defined in :numref:`sec_language_model`rr+Ncs||Srr)pos)rv num_stepsrrrgXsz"seq_data_iter_random..datacsg|] }|qSrrrprrrrrsasz(seq_data_iter_random..csg|]}|dqSr+rrzrrrrsbs)randomrandintr4r/rrfrtensor) rvrbryZ num_subseqsZinitial_indicesr(rqZinitial_indices_per_batchr1r8r)rvrgryrseq_data_iter_randomHs rc cstd|}t||d||}t||||}t||d|d|}||d||d}}|jd|}td|||D]>}|dd|||f} |dd|||f} | | fVqdS)zpGenerate a minibatch of subsequences using sequential partitioning. Defined in :numref:`sec_language_model`rr+rEN)r|r}r4rr~rVrUr) rvrbryoffset num_tokensZXsZYsr(rqr1r8rrrseq_data_iter_sequentiales rc@s eZdZdZddZddZdS) SeqDataLoaderz"An iterator to load sequence data.cCs:|rtj|_ntj|_t|\|_|_|||_|_dS)z'Defined in :numref:`sec_language_model`N) rr data_iter_fnrrwrvrtrbry)rArbryuse_random_iterrurrrrBws  zSeqDataLoader.__init__cCs||j|j|jSr)rrvrbryr@rrr__iter__szSeqDataLoader.__iter__N)rMrNrOrPrBrrrrrrus r'cCst||||}||jfS)zpReturn the iterator and the vocabulary of the time machine dataset. Defined in :numref:`sec_language_model`)rrt)rbryrrurrrrload_data_time_machinesrc@s(eZdZdZddZddZddZdS) RNNModelScratchz&An RNN Model implemented from scratch.cCs.|||_|_|||||_|||_|_dS)z$Defined in :numref:`sec_rnn_scratch`N) vocab_size num_hiddensr` init_state forward_fn)rArrr get_paramsrrrrrrBszRNNModelScratch.__init__cCs t|j|j}||||jSr)rone_hotTrrr`rAr1staterrr__call__szRNNModelScratch.__call__cCs|||j|Sr)rr)rArbrrrr begin_stateszRNNModelScratch.begin_stateN)rMrNrOrPrBrrrrrrrsrc s|jdd}|dgfdd}|ddD]"}|||\}}|q6t|D]0}|||\}}t|jdddqbdfd d DS) zYGenerate new characters following the `prefix`. Defined in :numref:`sec_rnn_scratch`r+rbrrcsttjdgddS)NrEr)r+r+)rrVr~r)routputsrrrszpredict_ch8..Nrcsg|]}j|qSrrjrorsrrrsszpredict_ch8..)rrFrrnrrVr) prefixZ num_predsrrtrrZ get_inputr;rr)rrrtr predict_ch8s rcCsrt|tjr$dd|D}n|j}ttdd|D}||krn|D]}|j dd||9<qNdS)z.css|]}|jdVqdS)r\N)r_rHrrrr sz grad_clipping..N) r.rBlockrrr`mathsqrtrHr_)rthetar`Znormrcrrr grad_clippings rc Cs dt}}td}|D]\} } |dks0|rF|j| jd|d}n|D] } | qJ| jd} | || |} } t "|| |\} }|| |  }W5QRX| t |d|dd||t| t| qt|d|d|d|fS)z`Train a model within one epoch (defined in Chapter 8). Defined in :numref:`sec_rnn_scratch`Nr\rrrEr+rb)rr=rrrUdetachrrVrrrmeanrrrrrexprG)rrrrrrrr'rr1r8sr;r]rrrrtrain_epoch_ch8s"        rcs tj}tjdddgd|gd}ttjrhjdt ddt  d d ifd d } nfd d } fdd } t |D]<} t ||| |\} } | dddkr|| d| gqtd| dd| ddtt| dt| ddS)zOTrain a model (defined in Chapter 8). Defined in :numref:`sec_rnn_scratch`rZ perplexityr )r r!rr"TrQ)rrrrdrcs |Sr)rr)r&rrrrTztrain_ch8..cstj|Sr)rrdr`r)rarrrrrTcst|dS)N2)r)r)rrrtrrrrTr+rz perplexity r ,  tokens/sec on ztime travellerZ travellerN)rrr$rrr.rr"rNormalrrrrrrr%)rrrtrarrrrrrZpredictrZpplZspeedr)rrarr&rtr train_ch8s<    " rcs0eZdZdZfddZddZddZZS)RNNModelz8The RNN model. Defined in :numref:`sec_rnn-concise`c s.tt|jf|||_||_t||_dSr)r/rrBr rr Densedense)rAZ rnn_layerrr9r:rrrBszRNNModel.__init__cCs@t|j|j}|||\}}||d|jd}||fS)NrE)rrrrr rrVrU)rAinputsrr1r8outputrrrr>szRNNModel.forwardcOs|jj||Sr)r r)rArr9rrrrszRNNModel.begin_state)rMrNrOrPrBr>rr@rrr:rrs  rz fra-eng.zipZ(94646ad1522d915e7b0f9296181140edcf86a4f5fra-engc Cs>td}ttj|dd}|W5QRSQRXdS)zRLoad the English-French dataset. Defined in :numref:`sec_machine_translation`rzfra.txtrN)rrrrrrr)rrrrr read_data_nmts rcsBddddddfddtD}d|S) zXPreprocess the English-French dataset. Defined in :numref:`sec_machine_translation`cSs|tdko|dkS)Nz,.!?rBset)rNZ prev_charrrrno_spacesz preprocess_nmt..no_spaceu rB cs6g|].\}}|dkr.||dr.d|n|qS)rr+rBr)rprqrNrtextrrrssz"preprocess_nmt..r)replacerFryr)routrrrpreprocess_nmts  rcCsxgg}}t|dD]V\}}|r0||kr0qp|d}t|dkr||dd||ddq||fS)zVTokenize the English-French dataset. Defined in :numref:`sec_machine_translation`r r\rrBr+)ryrr4rF)rrYsourcetargetrqrHpartsrrr tokenize_nmts    rcCstttjdd|Ddd|Dg\}}}tj|tj||djD]}|dqTtj|dS)z[Plot the histogram for list length pairs. Defined in :numref:`sec_machine_translation`cSsg|] }t|qSrr4rprrrrrs2sz+show_list_len_pair_hist..r+rN) rrrhistr r!patchesZ set_hatchr)rr r!ZxlistZylistrrpatchrrrshow_list_len_pair_hist,s    rcCs.t||kr|d|S||g|t|S)zLTruncate or pad sequences. Defined in :numref:`sec_machine_translation`Nr)rHryZ padding_tokenrrr truncate_pad9s  rcsdfdd|D}fdd|D}tfdd|D}tt|dktjd}||fS)zrTransform text sequences of machine translation into minibatches. Defined in :numref:`subsec_mt_data_loading`csg|] }|qSrrrrsrrrsEsz#build_array_nmt..csg|]}|dgqS)rrrsrrrsFscsg|]}t|dqS)rrryrtrrrsGs rr+)rr~rrint32)rJrtryrJ valid_lenrrrbuild_array_nmtAsrXcCstt}t||\}}tj|ddddgd}tj|ddddgd}t|||\}} t|||\} } || | | f} t| |} | ||fS)zuReturn the iterator and the vocabularies of the translation dataset. Defined in :numref:`subsec_mt_data_loading`r\rrrbrc)rrrrrQrrm)rbryrYrrr src_vocab tgt_vocabZ src_arrayZ src_valid_lenZ tgt_arrayZ tgt_valid_lenrjrrrr load_data_nmtMs   rcs(eZdZdZfddZddZZS)Encoderz@The base encoder interface for the encoder-decoder architecture.c stt|jf|dSr)r/rrBrAr9r:rrrB_szEncoder.__init__cGstdSrNotImplementedError)rAr1rrrrr>bszEncoder.forwardr?rrr:rr]s rcs0eZdZdZfddZddZddZZS)DecoderznThe base decoder interface for the encoder-decoder architecture. Defined in :numref:`sec_encoder-decoder`c stt|jf|dSr)r/rrBrr:rrrBiszDecoder.__init__cGstdSrr)rA enc_outputsrrrrrlszDecoder.init_statecCstdSrrrrrrr>oszDecoder.forward)rMrNrOrPrBrr>r@rrr:rres rcs(eZdZdZfddZddZZS)EncoderDecoderzbThe base class for the encoder-decoder architecture. Defined in :numref:`sec_encoder-decoder`c s"tt|jf|||_||_dSr)r/rrBencoderdecoder)rArrr9r:rrrBvszEncoderDecoder.__init__cGs.|j|f|}|jj|f|}|||Sr)rrr)rAenc_Xdec_Xrr dec_staterrrr>{szEncoderDecoder.forwardr?rrr:rrrs rcs*eZdZdZdfdd ZddZZS)Seq2SeqEncoderzXThe RNN encoder for sequence to sequence learning. Defined in :numref:`sec_seq2seq`rc s6tt|jf|t|||_tj|||d|_dS)N)dropout)r/rrBr Embedding embeddingr ZGRU)rArZ embed_sizer num_layersrr9r:rrrBszSeq2SeqEncoder.__init__cGsF||}|dd}|jj|jd|jd}|||\}}||fS)Nrr+r)rswapaxesr rrUr)rAr1rrrrrrr>s   zSeq2SeqEncoder.forward)rr?rrr:rrsrcs eZdZdZfddZZS)MaskedSoftmaxCELosszXThe softmax cross-entropy loss with masks. Defined in :numref:`sec_seq2seq_decoder`cs:tjt|dd}tj||ddd}tt||||S)NrErTr+)r expand_dims ones_liker sequence_maskr/rr>)rArlabelrweightsr:rrr>szMaskedSoftmaxCELoss.forward)rMrNrOrPr>r@rrr:rrsrc s|jtddt|dd|i}t}tjddd|gd}t |D]} t } t d } |D]} fd d | D\} }}}t j |d g|jd ddd}t||ddddfgd}t"|| ||\}}||||}W5QRX|t|d|}||| ||qh| ddd krL|| d| d | dfqLtd| d | ddd| d| ddtdS)zUTrain a model for sequence to sequence. Defined in :numref:`sec_seq2seq_decoder`T)rrZadamrrrr)r r!r"r\csg|]}|qSr)rrpr:rrrrssz!train_seq2seq..rrrrEr+Nrrrr r)r"rr#rrrrrrrr=rrrJrUrVconcatrrrrrHrrrrGr%)rrrarrrr&rrrr'rbatchr1Z X_valid_lenr8Z Y_valid_lenZbosZ dec_inputZY_hatrrrrrr train_seq2seqsD        rcCs ||d|dg}tjt|g|d}t|||d}tjtj||ddd} || |} |j | |} tjtj|dg|ddd} gg} }t |D]f}| | | \}} |j dd} | j ddd }|r||j j||dkrq | |qd|| |fS) zPPredict for sequence to sequence. Defined in :numref:`sec_seq2seq_training`rBrrrrrrr\r)rFrrrJr4rrrrrrrrsqueezeritemrFattention_weightsrrm)rZ src_sentencerrryrZsave_attention_weightsZ src_tokensZ enc_valid_lenrrrrZ output_seqZattention_weight_seqrr8rrrrpredict_seq2seqs(     rc Cs"|d|d}}t|t|}}ttdd||}td|dD]}dtt} } t||dD]&} | d || | |d7<qrt||dD]L} | d || | |dkr| d7} | d || | |d8<q|t | ||dt d|9}qN|S)z@Compute the BLEU. Defined in :numref:`sec_seq2seq_training`rBrr+r) rr4rrminrrqr rnrpow) Zpred_seqZ label_seqkZ pred_tokensZ label_tokensZlen_predZ len_labelscorerZ num_matchesZ label_subsrqrrrbleus$$&rrrRedscCst|jd|jd}}tjj|||dddd\}} tt| |D]v\} \} } tt| | D]Z\} \}}|jt||d}| |dkr| || dkr| ||rb| || qbqH|j || ddd S) zGShow heatmaps of matrices. Defined in :numref:`sec_attention-cues`rr+TF)rZsharexZshareyr)cmapg333333?)rshrinkN) rrrUrrwryr6rzr{rrr|Zcolorbar)Zmatricesr r!rrrr~rrrrqZrow_axesZ row_matricesrrmatrixZpcmrrr show_heatmapss"     rcCsp|dkrt|S|j}|jdkr2||d}n |d}tj|d|d|dddd}t||SdS)zyPerform softmax operation by masking elements on the last axis. Defined in :numref:`sec_attention-scoring-functions`Nr+rETg.)valuer)rZsoftmaxrUr*repeatrVr)r1 valid_lensrUrrrmasked_softmax s   rcs(eZdZdZfddZddZZS)AdditiveAttentionzMAdditive attention. Defined in :numref:`sec_attention-scoring-functions`c sXtt|jf|tj|ddd|_tj|ddd|_tjdddd|_t||_ dS)NFuse_biasrxr+) r/rrBr rW_kW_qw_vDropoutr)rArrr9r:rrrB s zAdditiveAttention.__init__cCsp||||}}tj|ddtj|dd}t|}tj||dd}t|||_t | |j|S)Nr\rr+rE) r r rrtanhrr rrr batch_dotr)rAquerieskeysrrfeaturesscoresrrrr>)s  zAdditiveAttention.forwardr?rrr:rrs rcs*eZdZdZfddZdddZZS)DotProductAttentionzQScaled dot product attention. Defined in :numref:`subsec_additive-attention`c s"tt|jf|t||_dSr)r/rrBr rrrArr9r:rrrB?szDotProductAttention.__init__NcCsD|jd}tj||ddt|}t|||_t||j|S)NrET)Z transpose_b)rUrrrrrrr)rArrrrdrrrrr>Hs  zDotProductAttention.forward)Nr?rrr:rr;s rcs,eZdZdZfddZeddZZS)AttentionDecoderz[The base attention-based decoder interface. Defined in :numref:`sec_seq2seq_attention`c stt|jf|dSr)r/rrBrr:rrrBSszAttentionDecoder.__init__cCstdSrrr@rrrrVsz"AttentionDecoder.attention_weights)rMrNrOrPrBrorr@rrr:rrOs rcs*eZdZdZdfdd ZddZZS)MultiHeadAttentionzGMulti-head attention. Defined in :numref:`sec_multihead-attention`Fc sptt|jf|||_t||_tj||dd|_ tj||dd|_ tj||dd|_ tj||dd|_ dS)NFr ) r/rrB num_headsrr attentionr rr r W_vW_o)rArrrr r9r:rrrB^s zMultiHeadAttention.__init__cCstt|||j}t|||j}t|||j}|dk rN|j|jdd}|||||}t||j}||S)Nrr) transpose_qkvr rr rrrtranspose_outputr)rArrrrrZ output_concatrrrr>hs zMultiHeadAttention.forward)Fr?rrr:rrZs rcCsF||jd|jd|d}|dddd}|d|jd|jdS)zuTransposition for parallel computation of multiple attention heads. Defined in :numref:`sec_multihead-attention`rr+rEr\rrVrU transposer1rrrrrs rcCsF|d||jd|jd}|dddd}||jd|jddS)z[Reverse the operation of `transpose_qkv`. Defined in :numref:`sec_multihead-attention`rEr+r\rrr r"rrrrsrcs*eZdZdZdfdd ZddZZS)PositionalEncodingzYPositional encoding. Defined in :numref:`sec_self-attention-and-positional-encoding`c stt|t||_td||f|_t | ddt dt d|d|}t ||jdddddddf<t ||jdddddddf<dS)Nr+rErrr\)r/r#rBr rrrr ParangerVrpowersincos)rArrmax_lenr1r:rrrBs $zPositionalEncoding.__init__cCs8||jddd|jdddf|j}||SrR)r%rUrrrrAr1rrrr>s.zPositionalEncoding.forward)r$r?rrr:rr#s r#cs(eZdZdZfddZddZZS)PositionWiseFFNzLPositionwise feed-forward network. Defined in :numref:`sec_transformer`c s8tt|jf|tj|ddd|_tj|dd|_dSNFr<)rx activation)rx)r/r,rBr rdense1dense2)rAffn_num_hiddensZffn_num_outputsr9r:rrrBs zPositionWiseFFN.__init__cCs|||Sr)r0r/r+rrrr>szPositionWiseFFN.forwardr?rrr:rr,s r,cs(eZdZdZfddZddZZS)AddNormz^Residual connection followed by layer normalization. Defined in :numref:`sec_transformer`c s,tt|jf|t||_t|_dSr)r/r2rBr rr LayerNormlnrr:rrrBs zAddNorm.__init__cCs||||Sr)r4rr=rrrr>szAddNorm.forwardr?rrr:rr2s r2cs*eZdZdZdfdd ZddZZS) EncoderBlockzDTransformer encoder block. Defined in :numref:`sec_transformer`Fc sHtt|jf|t|||||_t||_t|||_ t||_ dSr) r/r5rBrrrr2addnorm1r,ffnaddnorm2)rArr1rrr r9r:rrrBs  zEncoderBlock.__init__c Cs*|||||||}||||Sr)r6rr8r7)rAr1rr8rrrr>szEncoderBlock.forward)Fr?rrr:rr5s r5cs*eZdZdZdfdd ZddZZS)TransformerEncoderz>Transformer encoder. Defined in :numref:`sec_transformer`Fc shtt|jf|||_t|||_t|||_ t |_ t |D]} |j t|||||qFdSr)r/r9rBrr rrrr# pos_encoding Sequentialblksrrr5) rArrr1rrrr r9rr:rrrBs   zTransformerEncoder.__init__cGs`|||t|j}dgt|j|_t|jD]"\}}|||}|j j j|j|<q8|Sr) r:rrrrr4r<rryr)rAr1rrrqblkrrrr>s zTransformerEncoder.forward)Fr?rrr:rr9s r9cCs"tjj|||tddddS)Nz->)Z arrowstyle)xyxytextZ arrowprops)rrr3annotatedict)rr>r?rrrr@sr@c Csd\}}}}||fg}t|D]F}|r@||||||\}}}}n|||||\}}}}|||fqtd|ddt|ddt|d|S)zlOptimize a 2D objective function with a customized trainer. Defined in :numref:`subsec_gd-learningrate`)rrzepoch r+z, x1: rz, x2: )rrFrr) r&stepsZf_gradx1x2s1s2resultsrqrrrtrain_2ds   *rKcCs|ttjjt|d ddittdddtddd\}}tjj|||||dd tjd tj d d S)zdShow the trace of 2D variables during optimization. Defined in :numref:`subsec_gd-learningrate`-ocolorz#ff7f0eg?皙?gz#1f77b4)colorsrFrGN)rL) rrrr7r6meshgridr&Zcontourr r!)rrJrFrGrrr show_trace_2d s  rRzairfoil_self_noise.datZ(76e5be1548fd8222e5074cf0faae75edff8cf93fairfoilrcCsvtjtdtjdd}||jdd|jdd}tj|d|ddf|d|dff|dd }||jd d fS) $Defined in :numref:`sec_minibatches`rSr)r delimiterrrNrETrkr+) rZ genfromtxtrrfloat32rstdrmrU)rbrrgrrrr get_data_ch11s "rZr\c sTtjjd|dfdtdfddtj}}tjddd|gd d gd }dt} } t |D]} |D]\} } t ||| |  }W5QRX| |g||| | jd7} | d dkr|| || | jdt|t|||f| q|qttd |jdddd| dd| |jdfS)rUrQr+)rrcst|Sr)rr[r0rXrWrrr,rTztrain_ch11..rrr)\(?ffffff?r r!r"r#loss: rErr sec/epoch)rr|rSr Z attach_gradrr^rr=rrrrrrUrGrr4rr?rr8rIrK)Z trainer_fnstates hyperparamsrZ feature_dimrrrrrr'rr1r;rrr[r train_ch11$s2      (rdc CsDt}|td|tjddt| ||}tj }t j ddd|gddgd }dt }} t|D]} |D]\} } t||| | } W5QRX| || jd|| jd7}|d dkr~| ||| jdt|t |||f| q~qvtd |jdd d d| d ddS)rUr+rQ)sigmarrrr\r]r^r_r`rErrraN)r r;rrr"rrrrrrZL2Lossrrr=rrrrrrUrGr4rr?rr8rI)Ztr_namercrrrr&rrrr'rr1r;rrrrtrain_concise_ch11@s0      rfc@s*eZdZdZd ddZddZddZd S) BenchmarkzFor measuring running time.DonecCs ||_dS)z"Defined in :numref:`sec_hybridize`N) description)rArirrrrB[szBenchmark.__init__cCst|_|Sr)rr=r'r@rrr __enter___s zBenchmark.__enter__cGs"t|jd|jdddS)Nz: z.4fz sec)rrir'rGrrrr__exit__cszBenchmark.__exit__N)rh)rMrNrOrPrBrjrkrrrrrgYs rgcCs4|jd|jdksttj||tj||fS)zPSplit `X` and `y` into multiple devices. Defined in :numref:`sec_multi_gpu`r)rUrrutilssplit_and_loadr1r;r rrr split_batchfs  rocCs~ddd}t}|tjdddddttd||dd d d |d d |d d |dd |tt||S)zTA slightly modified ResNet-18 model. Defined in :numref:`sec_multi_gpu_concise`FcSsNt}t|D]8}|dkr8|s8|tj|dddq|t|q|S)NrTr\)r8r.)r r;rrrr+)r7Z num_residuals first_blockr=rqrrr resnet_blockrs   zresnet18..resnet_block@rr+)r,r.r-r<r\T)rpi)F)r r;rr0r4Z ActivationZGlobalAvgPool2Dr) num_classesrqrrrrresnet18ns rvc std}td}|D]L\}}||||\}}fdd|D} |tddt| |D|j q&|d|dS)zrCompute the accuracy for a model on a dataset using multiple GPUs. Defined in :numref:`sec_multi_gpu_concise`rr\csg|] }|qSrrrpZX_shardrrrrssz*evaluate_accuracy_gpus..css"|]\}}tt||VqdSr)rrrrpZ pred_shardZy_shardrrrrsz)evaluate_accuracy_gpus..r+) r/rrrrrrrHr6r) rrsplit_fr rrrtX_shardsy_shards pred_shardsrrxrevaluate_accuracy_gpuss   r~c s||||\}}t0fdd|D} fddt| |D} W5QRX| D] } | qR|j|jdddtdd| D} tdd t| |D} | | fS) zqTrain for a minibatch with mutiple GPUs (defined in Chapter 13). Defined in :numref:`sec_image_augmentation`csg|] }|qSrrrwrxrrrssz$train_batch_ch13..csg|]\}}||qSrrryrrrrssrT)Zignore_stale_gradcSsg|]}t|qSr)rrHrrrrrsscss|]\}}t||VqdSr)rrryrrrrsz#train_batch_ch13..)rrr6rrrUrH)rrrtrr&r rzr{r|r}lsrZtrain_loss_sumZ train_acc_sumrrrrtrain_batch_ch13s   rc Cs~tt|}} tjdd|gddgdddgd} t|D]} td} t|D]\} \}}|t|||||||\}}| |||j d|j | | d| d dks| | dkrP| | | d| | d| d | d| d d fqPt |||}| | dd d |fq:td | d| d dd| d| d dd|dt| d ||ddt|d S)ziTrain a model with mutiple GPUs (defined in Chapter 13). Defined in :numref:`sec_image_augmentation`rr+rrrrrrrr\rNrrrrr r!)rr=r4rrrryr?rrrUrrGr~rrHr%)rrrrr&rr rzr'r(rrrrqrrtraccrrrr train_ch13s:   8rz hotdog.zipZ(fba480ffa8aa7e0febbb511d181409f899b9baa5Zhotdogc Cs|dddf|dddf|dddf|dddff\}}}}||d}||d}||}||}tj||||fdd}|S)zeConvert from (upper-left, lower-right) to (center, width, height). Defined in :numref:`sec_bbox`Nrr+r\rrErrstack) boxesrFy1rGy2cxcyrWrrrrbox_corner_to_centersD  rc Cs|dddf|dddf|dddf|dddff\}}}}|d|}|d|}|d|}|d|}tj||||fdd}|S) zeConvert from (center, width, height) to (upper-left, lower-right). Defined in :numref:`sec_bbox`Nrr+r\rrrErr) rrrrWrrFrrGrrrrbox_center_to_cornersD    rcCs<tjj|d|df|d|d|d|dd|ddS)zMConvert bounding box to matplotlib format. Defined in :numref:`sec_bbox`rr+r\rF)r>widthheightfillZ edgecolorZ linewidth)rrZ Rectangle)bboxrMrrr bbox_to_rectsrc Cs|jdd\}}|jt|t|}}}||d}tj||d} tj||d} d\} } d|} d|}tj||d| | }tj||d| |}t||\}}|d|d}}t | t | d|dt | ddf||}t | t | d|dt | ddf}t t | | ||fj ||dfd }tj ||||gdd j|dd }||}tj|dd S) zhGenerate anchor boxes with different shapes centered on each pixel. Defined in :numref:`sec_anchor`rDNr+r)rrrNrErr\r)rUrr4rr~r&rQrVr concatenatertilerrrr)rgsizesratiosZ in_heightZin_widthrZ num_sizesZ num_ratiosZboxes_per_pixelZ size_tensorZ ratio_tensorZoffset_hZoffset_wZsteps_hZsteps_wZcenter_hZcenter_wZshift_xZshift_yrWrZanchor_manipulationsZout_gridrrrrmultibox_priorsF  rc Csddd}||}||dddddg}t|D]\}}||t|}tt||}|||r.t||kr.|d kr~d nd } |j|jd |jd ||d d d| t|d ddq.dS)z9Show bounding boxes. Defined in :numref:`sec_anchor`NcSs&|dkr|}nt|ttfs"|g}|Sr)r.r/rh)objZdefault_valuesrrr make_lists zshow_bboxes..make_listrXgrmcrWrrr+center )Z facecolorZlw)vahaZfontsizerMr)N) ryr4rrr{Z add_patchrr>rA) rZbboxesrtrPrrqrrMrectZ text_colorrrr show_bboxess    rc Csdd}||}||}t|dddddf|ddddf}t|dddddf|ddddf}||jdd}|dddddf|dddddf}|dddf||} || S)zgCompute pairwise IoU across two lists of anchor or bounding boxes. Defined in :numref:`sec_anchor`cSs@|dddf|dddf|dddf|dddfS)Nr\rrr+r)rrrrr3szbox_iou..Nr\r)rr+)rmaximumminimumclip) Zboxes1Zboxes2Zbox_areaZareas1Zareas2Zinter_upperleftsZinter_lowerrightsZintersZ inter_areasZ union_areasrrrbox_iou/s..,rrcCs|jd|jd}}t||}tj|fdtj|d}tj|ddtj|dd}} t|dkd} | |dk} | || <t|fd} t|fd} t|D]R}t|}|| d}|| d}|||<| |dd|f<| ||ddf<q|S) z`Assign closest ground-truth bounding boxes to anchor boxes. Defined in :numref:`sec_anchor`rrErrr+rrrN) rUrrfullrmaxrnonzerorr)Z ground_truthanchorsr iou_threshold num_anchorsZ num_gt_boxesZjaccardanchors_bbox_mapZmax_iousrlZanc_iZbox_jZ col_discardZ row_discardrZmax_idxZbox_idxZanc_idxrrrassign_anchor_to_bboxCs"    rư>c Cst|}t|}d|ddddf|ddddf|ddddf}dt||ddddf|ddddf}tj||gdd}|S)zXTransform for anchor box offsets. Defined in :numref:`subsec_labeling-anchor-boxes`rNr\rr+r)rrlogr)r assigned_bbepsZc_ancZ c_assigned_bbZ offset_xyZ offset_whrrrr offset_boxes^s   @6rcCsh|jd|d}}ggg}}}|j|jd}}t|D]}||ddddf} t| ddddf||} ttj| dkdddd} t j |tj |d} t j |d ftj |d} t | dkd}| |}| |dfdd| |<| |ddf| |<t|| | }||d|| d|| q@t |}t |} t |} || | fS) zlLabel anchor boxes using ground-truth bounding boxes. Defined in :numref:`subsec_labeling-anchor-boxes`rNr+rEr)r+rrrr)rUrrrrrrrrrr rrXrrrFrVr)rrtrbZ batch_offsetZ batch_maskZbatch_class_labelsrrrqrrZ bbox_maskZ class_labelsrZ indices_trueZbb_idxrZ bbox_offsetrrrmultibox_targetisB     rcCst|}|ddddf|ddddfd|ddddf}t|ddddfd|ddddf}tj||fdd}t|}|S)z{Predict bounding boxes based on anchor boxes with predicted offsets. Defined in :numref:`subsec_labeling-anchor-boxes`Nr\rrr+r)rrrrr)r offset_predsZancZ pred_bbox_xyZ pred_bbox_whZ pred_bboxZpredicted_bboxrrroffset_inverses  @2 rcCs|ddd}g}|jdkr|d}|||jdkr>qt||ddfdd||ddddfddd}t||kd}||d}qtj|tj|j dS)zrSort confidence scores of predicted bounding boxes. Defined in :numref:`subsec_predicting-bounding-boxes-nms`NrErr+rr) argsortrrFrrVrrrJrr)rrrBkeeprqZiouZindsrrrnmss   rQ%z?cCst|j|jd}}tj|dd}|jd|jd}}g} t|D]$} || || dd} } t| dddt| ddd} }t|| }t || |}tj |tj |d}t ||f}tj|d d \}}||dk}t ||f}d||<||d }| |||} }| |k}d||<d| || |<t j tj|ddtj| dd|fdd}| |qBt | S) ztPredict bounding boxes using non-maximum suppression. Defined in :numref:`subsec_predicting-bounding-boxes-nms`rrr+r\rErNrT)Z return_countsrX)rrUrrrrVrrrrr&rrruniquerrrFr)Z cls_probsrrZ nms_thresholdZ pos_thresholdrrbrurrrqZcls_probZ offset_predconfZclass_idZ predicted_bbrZall_idxcombinedrcountsZnon_keepZ all_id_sortedZ below_min_idxZ pred_inforrrmultibox_detections8*     rzbanana-detection.zipZ(5de26c8fce5ccdea9f91267273464dc968d20d72banana-detectionc Cstd}tj||rdndd}t|}|d}gg}}|D]>\}}| t tj||rhdndd|| t |qH|t t |ddfS) zkRead the banana detection dataset images and labels. Defined in :numref:`sec_object-detection-dataset`rZ bananas_trainZ bananas_valz label.csvimg_nameimagesr+rt)rrrrrpdread_csvZ set_indexZiterrowsrFrimreadr/rrrJ)rkrZ csv_fnamecsv_datartargetsrrrrrread_data_bananass$    rc@s(eZdZdZddZddZddZdS) BananasDatasetzqA customized dataset to load the banana detection dataset. Defined in :numref:`sec_object-detection-dataset`cCs6t|\|_|_tdtt|j|r*dnddS)Nread z training examplesz validation examples)rrrtrr%r4)rArkrrrrBs zBananasDataset.__init__cCs$|j|dddd|j|fS)NrXr\rr+)rrr!rtrrrrrszBananasDataset.__getitem__cCs t|jSrr4rr@rrrr,szBananasDataset.__len__NrMrNrOrPrBrr,rrrrrsrcCs4tjjtdd|dd}tjtdd|}||fS)zYLoad the banana detection dataset. Defined in :numref:`sec_object-detection-dataset`TrWreF)rrgrir)rbrZval_iterrrrload_data_bananassrzVOCtrainval_11-May-2012.tarZ(4e443f8a2eca6b1dac8a6c57641b67dd40621a49voc2012c Cstj|dd|rdnd}t|d}|}W5QRXgg}}t|D]L\}}|t tj|d|d|t tj|d|d qN||fS) zZRead all VOC feature and label images. Defined in :numref:`sec_semantic_segmentation`Z ImageSetsZ Segmentationz train.txtzval.txtrZ JPEGImagesz.jpgZSegmentationClassz.png) rrrrrrryrFrr) voc_dirrkZ txt_fnamerrrrtrqrrrrread_voc_imagess$     rrsrr backgroundZ aeroplaneZbicycleZbirdZboatZbottleZbusZcarcatZchairZcowZ diningtableZdogZhorseZ motorbikepersonz potted plantZsheepZsofarz tv/monitorcCsDtd}ttD],\}}|||dd|dd|d<q|S)zoBuild the mapping from RGB to class indices for VOC labels. Defined in :numref:`sec_semantic_segmentation`irrtr+r\)rr ry VOC_COLORMAP)colormap2labelrqcolormaprrrvoc_colormap2labels rcCs^|tj}|dddddfd|dddddfd|dddddf}||S)zlMap any RGB values in VOC labels to their class indices. Defined in :numref:`sec_semantic_segmentation`Nrrtr+r\)rrr)rrrrrrvoc_label_indices$s  2rcCs,t|||f\}}tj|f|}||fS)z`Randomly crop both feature and label images. Defined in :numref:`sec_semantic_segmentation`)rZ random_cropZ fixed_crop)featurerrrrrrr voc_rand_crop-src@s8eZdZdZddZddZddZdd Zd d Zd S) VOCSegDatasetzaA customized dataset to load the VOC dataset. Defined in :numref:`sec_semantic_segmentation`cstdddg_tdddg_|_t||d\}}fdd |D_|_t _ t d t t jd dS) Ng ףp= ?gv/?gCl?gZd;O?gy&1?g?rWcsg|]}|qSr)normalize_imagerprr@rrrs>sz*VOCSegDataset.__init__..r examples)rrJrgb_meanrgb_std crop_sizerfilterrrtrrrr%r4)rArkrrrrtrr@rrB9s  zVOCSegDataset.__init__cCs|dd|j|jS)NrX)rrr)rArrrrrDszVOCSegDataset.normalize_imagecsfdd|DS)Ncs8g|]0}|jdjdkr|jdjdkr|qSrr+)rUr)rprr@rrrsHsz(VOCSegDataset.filter..r)rAr}rr@rrGszVOCSegDataset.filtercCs<t|j||j|f|j\}}|dddt||jfS)Nr\rr+)rrrtrr!rr)rArrrrrrrLs    zVOCSegDataset.__getitem__cCs t|jSrrr@rrrr,RszVOCSegDataset.__len__N) rMrNrOrPrBrrrr,rrrrr5s  rcCs`tdtjdd}t}tjjt d|||dd|d}tjjt d|||d|d}||fS) z_Load the VOC semantic segmentation dataset. Defined in :numref:`sec_semantic_segmentation`rZ VOCdevkitZVOC2012Tdiscard)rf last_batchrF)rr) rrrrrrrrgrir)rbrrrrrrrr load_data_vocUs&   rzkaggle_cifar10_tiny.zipZ(2068874e4b9a9f0fb07ebe0ad2b29754449ccacdZ cifar10_tinyc CsFt|d}|dd}W5QRXdd|D}tdd|DS)zcRead `fname` to return a filename to label dictionary. Defined in :numref:`sec_kaggle_cifar10`rr+NcSsg|]}|dqS),)rstriprrrrrrsnsz#read_csv_labels..css|]\}}||fVqdSrr)rprrrrrrosz"read_csv_labels..)rrIrA)rrrJrarrrread_csv_labelsgs rcCstj|ddt||dS)zQCopy a file into a target directory. Defined in :numref:`sec_kaggle_cifar10`TrN)rrshutilcopy)filename target_dirrrrcopyfileqsrc Cst|dd}tdt||}i}ttj |dD]}|| dd}tj |d|}t |tj |dd|||ks|||krt |tj |dd|| |dd||<qFt |tj |dd|qF|S) zgSplit the validation set out of the original training set. Defined in :numref:`sec_kaggle_cifar10`rEr+rrrtrain_valid_testZ train_validvalid)rqrrr most_commonrrfloorrlistdirrrrrr) rrtZ valid_ratiorZn_valid_per_labelZ label_countZ train_filerrrrrreorg_train_validxs*rc CsBttj|dD](}ttj|d|tj|dddqdS)ziOrganize the testing set for data loading during prediction. Defined in :numref:`sec_kaggle_cifar10`testrunknownN)rrrrr)rZ test_filerrr reorg_tests  rzkaggle_dog_tiny.zipZ(0cb91d09b814ecdc07b50f31f8dcad3e81d6a86dZdog_tinyzptb.zipZ(319d85e578af0cdc590547f26231e4e31cdf1e42ptbc CsDtd}ttj|d}|}W5QRXdd|dDS)z[Load the PTB dataset into a list of text lines. Defined in :numref:`sec_word2vec_data`rz ptb.train.txtcSsg|] }|qSrrMrGrrrrsszread_ptb..r)rrrrrrrr)rrraw_textrrrread_ptbs rcsLfdd|D}t|tfddfdd|DfS)zKSubsample high-frequency words. Defined in :numref:`sec_word2vec_data`csg|]}fdd|DqS)csg|]}|jkr|qSr)rirgrsrrrss(subsample...rrGrsrrrsszsubsample..cs"tddtd|kS)Nrr+g-C6?)r|uniformrrrO)rdrrrrs zsubsample..keepcsg|]}fdd|DqS)csg|]}|r|qSrrrgrrrrssrrrGrrrrss)rr[rHr) sentencesrtr)rdrrrtr subsamples   rc sgg}}|D]tdkr q|7}ttD]\}td|}tttd||tt|d|}|||fdd|Dq4q||fS)z_Return center words and context words in skip-gram. Defined in :numref:`sec_word2vec_data`r\r+rcsg|] }|qSrr)rprrHrrrssz,get_centers_and_contexts..) r4rr|r}r/rrremoverF)rvmax_window_sizecenterscontextsrqZ window_sizerlrrrget_centers_and_contextss    r c@s eZdZdZddZddZdS)RandomGeneratorz@Randomly draw among {1, ..., n} according to n sampling weights.cCs.ttdt|d|_||_g|_d|_dS)z&Defined in :numref:`sec_word2vec_data`r+rN)r/rr4 populationsampling_weights candidatesrq)rAr rrrrBszRandomGenerator.__init__cCsJ|jt|jkr,tj|j|jdd|_d|_|jd7_|j|jdS)Nrrrr+)rqr4rr|choicesr r r@rrrdrawszRandomGenerator.drawN)rMrNrOrPrBrrrrrr sr c sxfddtdtD}gt|}}|D]B}g}t|t||krh|} | |kr8|| q8||q0|S)zTReturn noise words in negative sampling. Defined in :numref:`sec_word2vec_data`csg|]}|dqS)g?)rmrordrtrrrssz!get_negatives..r+)rr4r rrF) all_contextsrtrdrr  all_negatives generatorr Z negativesnegrrr get_negativess    rc Cstdd|D}ggggf\}}}}|D]\}}}t|t|} ||g7}|||dg|| g7}|dg| dg|| g7}|dgt|dg|t|g7}q*tt|dt|t|t|fS)zpReturn a minibatch of examples for skip-gram with negative sampling. Defined in :numref:`sec_word2vec_data`css$|]\}}}t|t|VqdSrr)rprrrrrrrszbatchify..rr+rR)rr4rrVr~) rgr*rZcontexts_negativesmasksrtrrnegativecur_lenrrrbatchifys (rc st}tj|ddt|\}}fdd|D}t||\}}t|||} tj||| } tjj | |dt t d} | fS)zrDownload the PTB dataset and then load it into memory. Defined in :numref:`subsec_word2vec-minibatch-loading`rrbcsg|] }|qSrrrGrsrrrs sz!load_data_ptb..T)rfZ batchify_fnr) rrrQrr rrrgrhrirr) rbrZnum_noise_wordsrZ subsampledrdrvZ all_centersrrrlrrrsr load_data_ptbs6rzglove.6B.50d.zipZ(0b8703943ccdb6eb788e6f091b8946e82231bc4dz glove.6b.50dzglove.6B.100d.zipZ(cd43bfb07e44e6f27cbcc7bc9ae3d80284fdaf5az glove.6b.100dzglove.42B.300d.zipZ(b5116e234e9eb9076672cfeabf5469f3eec904fazglove.42b.300dz wiki.en.zipZ(c1816da3821ae9f43899be655002f6c723e91b88zwiki.enc@s0eZdZdZddZddZddZdd Zd S) TokenEmbeddingzToken Embedding.cCs2||\|_|_d|_ddt|jD|_dS)z!Defined in :numref:`sec_synonyms`rcSsi|]\}}||qSrrrYrrrrZ*sz+TokenEmbedding.__init__..N)_load_embeddingr_ idx_to_vec unknown_idxryr`)rAembedding_namerrrrB%s zTokenEmbedding.__init__c Csdgg}}t|}ttj|dd^}|D]R}|d}|ddd|ddD}}t|dkr0| || |q0W5QRXdgt|dg|}|t |fS) NrXzvec.txtrrBrcSsg|] }t|qSrr)rpelemrrrrs5sz2TokenEmbedding._load_embedding..r+) rrrrrrrrr4rFr~) rAr"r_r rrrHelemsrOrrrr-s     zTokenEmbedding._load_embeddingcs&fdd|D}jt|}|S)Ncsg|]}j|jqSr)r`rr!rgr@rrrs>sz.TokenEmbedding.__getitem__..)r rr~)rArarlZvecsrr@rr=s  zTokenEmbedding.__getitem__cCs t|jSrrfr@rrrr,CszTokenEmbedding.__len__N)rMrNrOrPrBrrr,rrrrr#s rcCsVdg|dg}dgt|d}|dk rN||dg7}|dgt|d7}||fS)z_Get tokens of the BERT input sequence and their segment IDs. Defined in :numref:`sec_bert`rr\Nr+r)tokens_atokens_brasegmentsrrrget_tokens_and_segmentsFs r*cs*eZdZdZdfdd ZddZZS) BERTEncoderz=BERT encoder. Defined in :numref:`subsec_bert_input_rep`r$c s|tt|jf|t|||_td||_t|_t |D]} |j t ||||dq@|j jdd||fd|_dS)Nr\T pos_embeddingr+)rU)r/r+rBr rtoken_embeddingsegment_embeddingr;r<rrrr5r`rr,) rArrr1rrrr*r9rr:rrrBVs   zBERTEncoder.__init__cCs^||||}||jj|jdddd|jdddf}|jD]}|||}qJ|S)Nrr+)r-r.r,rgrrUr<)rArar)rr1r=rrrr>ds 0  zBERTEncoder.forward)r$r?rrr:rr+Rsr+cs(eZdZdZfddZddZZS)MaskLMzWThe masked language model task of BERT. Defined in :numref:`subsec_bert_input_rep`c s^tt|jf|t|_|jtj|ddd|jt|jtj|dddSr-) r/r/rBr r;mlprrr3)rArrr9r:rrrBqs zMaskLM.__init__cCs`|jd}|d}|jd}td|}t||}|||f}|||df}||}|S)Nr+rEr)rUrVrr&rr0)rAr1pred_positionsZnum_pred_positionsrbZ batch_idxZmasked_X mlm_Y_hatrrrr>ys       zMaskLM.forwardr?rrr:rr/ms r/cs(eZdZdZfddZddZZS)NextSentencePredzOThe next sentence prediction task of BERT. Defined in :numref:`subsec_mlm`c s"tt|jf|td|_dS)Nr\)r/r3rBr rrrr:rrrBszNextSentencePred.__init__cCs ||Sr)rr+rrrr>szNextSentencePred.forwardr?rrr:rr3s r3cs,eZdZdZdfdd Zd ddZZS) BERTModelz4The BERT model. Defined in :numref:`subsec_nsp`r$csLtt|t||||||||_tj|dd|_t|||_ t |_ dS)Nr)r.) r/r4rBr+rr rhiddenr/mlmr3nsp)rArrr1rrrr*r:rrrBs zBERTModel.__init__Nc CsT||||}|dk r$|||}nd}|||dddddf}|||fSr)rr6r7r5)rArar)rr1Z encoded_Xr2 nsp_Y_hatrrrr>s "zBERTModel.forward)r$)NNr?rrr:rr4s r4)zHhttps://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zipZ(3c914d17d80b1459be871a5039ac23e752a53cbe wikitext-2c CsHtj|d}t|d}|}W5QRXdd|D}t||S)%Defined in :numref:`sec_bert-dataset`zwiki.train.tokensrcSs0g|](}t|ddkr|dqS)z . r\)r4rrErFrGrrrrssz_read_wiki..)rrrrrIr|rf)r file_namerrJ paragraphsrrr _read_wikis  r=cCs0tdkrd}ntt|}d}|||fS)r:rTF)r|choice)sentenceZ next_sentencer<is_nextrrr_get_next_sentences  rAc Csvg}tt|dD]\}t||||d|\}}}t|t|d|krPqt||\} } || | |fq|S)r:r+r)rr4rArr*rF) paragraphr<rtr*Znsp_data_from_paragraphrqr'r(r@rar)rrr_get_nsp_data_from_paragraphs  rCcCsdd|D}g}t||D]f}t||kr4qd}tdkrJd}n"tdkr`||}n t|j}|||<||||fq ||fS)r:cSsg|]}|qSrrrgrrrrssz'_replace_mlm_tokens..Ng?r)r|rfr4r>r_rF)racandidate_pred_positions num_mlm_predsrtmlm_input_tokenspred_positions_and_labelsZmlm_pred_positionZ masked_tokenrrr_replace_mlm_tokenss"       rIc Csg}t|D]\}}|dkrq ||q tdtt|d}t||||\}}t|ddd}dd|D}d d|D} ||||| fS) ,Defined in :numref:`subsec_prepare_mlm_data`)r%r&r+333333?cSs|dSrrrSrrrrrTz+_get_mlm_data_from_tokens..rVcSsg|] }|dqSrrrpvrrrrssz-_get_mlm_data_from_tokens..cSsg|] }|dqSr{rrNrrrrss)ryrFrroundr4rIr\) rartrErqrOrFrGrHr1Zmlm_pred_labelsrrr_get_mlm_data_from_tokenss$ rQc CsLt|d}ggg}}}ggg}}} g} |D]\} } } }}|tj| |dg|t| dd|tj|dg|t|dd|tjt| dd|tj| dg|t| dd|tjdgt| dg|t| dd| tj| dg|t| dd| t|q4|||||| | fS) rJrKrrrrrXrNr)rPrFrrJr4)examplesr*rtZmax_num_mlm_preds all_token_ids all_segmentsrall_pred_positionsall_mlm_weightsall_mlm_labels nsp_labelsZ token_idsr1Zmlm_pred_label_idsr)r@rrr_pad_bert_inputs sR      rZc@s(eZdZdZddZddZddZdS) _WikiTextDatasetrJcsdd|D}dd|D}tj|dddddgd _g}|D]}|t||j|q>fd d|D}t||j\____ _ _ _ dS) NcSsg|]}tj|ddqS)rLr)rrP)rprBrrrrs" s z-_WikiTextDataset.__init__..cSsg|]}|D]}|q qSrr)rprBr?rrrrs$ srrrDr%r&rcs&g|]\}}}t|j||fqSr)rQrt)rprar)r@r@rrrs. s ) rrQrtextendrCrZrTrUrrVrWrXrY)rAr<r*rrSrBrr@rrB s@  z_WikiTextDataset.__init__cCs<|j||j||j||j||j||j||j|fSr)rTrUrrVrWrXrYrrrrr7 sz_WikiTextDataset.__getitem__cCs t|jSr)r4rTr@rrrr,= sz_WikiTextDataset.__len__Nrrrrrr[ sr[cCsDt}tdd}t|}t||}tjj||d|d}||jfS)zNLoad the WikiText-2 dataset. Defined in :numref:`subsec_prepare_mlm_data`r9Tr) rrrr=r[rrgrirt)rbr*rrr< train_setrrrrload_data_wiki@ s   r^c Csggg} } } t||||||| D]\} }}}}}}|| ||d|\}}}||d|f|d|d}||d}|||}|}| || || ||tq$| | | fS)z)Defined in :numref:`sec_bert-pretraining`rErRg:0yE>)r6rVrHrrFrZwaitall)rrrZtokens_X_shardsZsegments_X_shardsZvalid_lens_x_shardsZpred_positions_X_shardsZmlm_weights_X_shardsZ mlm_Y_shardsZ nsp_y_shardsZmlm_lsZnsp_lsrZtokens_X_shardZsegments_X_shardZvalid_lens_x_shardZpred_positions_X_shardZmlm_weights_X_shardZ mlm_Y_shardZ nsp_y_shardrr2r8Zmlm_lZnsp_lrrr_get_batch_loss_bertL sD       r_)z>http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gzZ(01ada507287d82875905620988597833ad4e0903aclImdbc Csgg}}dD]}tj||r"dnd|}t|D]Z}ttj||d<}|ddd}||||dkr~d nd W5QRXq4q||fS) z_Read the IMDb review dataset text sequences and labels. Defined in :numref:`sec_sentiment`)rxrrrrzutf-8rrrxr+r) rrrrrrdecoderrF) rrkrgrtrZ folder_namefilerZreviewrrr read_imdbn s  $rcc stdd}t|d}t|d}tj|ddd}tj|ddd}tj|ddtfd d |D}tfd d |D}t||d f|} tj||d f|dd } | | fS)zlReturn data iterators and the vocabulary of the IMDb review dataset. Defined in :numref:`sec_sentiment`r`TFrrLrrrcs"g|]}t|dqSrrrrGrrrrs s z"load_data_imdb..cs"g|]}t|dqSrrerGrrrrs s r+rW)rrrcrPrQrrJrm) rbryr train_data test_dataZ train_tokensZ test_tokensZtrain_featuresZ test_featuresrrrrrload_data_imdb} s"   rhcCsDtj||td}tj||dddd}|dkr@dSdS)zUPredict the sentiment of a text sequence. Defined in :numref:`sec_sentiment_rnn`rr+rErpositiver)rrJrrr rrV)rrtsequencerrrrpredict_sentiment srk)z3https://nlp.stanford.edu/projects/snli/snli_1.0.zipZ(9fcde07509c7e87ec61c640c1b2753d9041758e4SNLIc sddddddtj||r$dnd}t|d  }d d |dd D}W5QRXfd d |D}fdd |D}fdd |D}|||fS)zRead the SNLI dataset into premises, hypotheses, and labels. Defined in :numref:`sec_natural-language-inference-and-dataset`cSs2tdd|}tdd|}tdd|}|S)Nz\(rz\)z\s{2,}rB)rCrDrE)rrrr extract_text szread_snli..extract_textrr+r\) entailment contradictionneutralzsnli_1.0_train.txtzsnli_1.0_test.txtrcSsg|]}|dqS)rrMrprowrrrrs szread_snli..Ncs$g|]}|dkr|dqSrrrqrm label_setrrrs s cs$g|]}|dkr|dqS)rr\rrqrsrrrs s cs$g|]}|dkr|dqSrMrrq)rtrrrs s )rrrrrI)rrkr;rrowspremises hypothesesrtrrsr read_snli s  $rxc@s2eZdZdZd ddZddZddZd d ZdS) SNLIDatasetzsA customized dataset to load the SNLI dataset. Defined in :numref:`sec_natural-language-inference-and-dataset`NcCs||_t|d}t|d}|dkrDtj||ddgd|_n||_|||_|||_t |d|_ t dt t |jddS) Nrr+rrrr\rr)ryrrPrQrt_padrvrwrrJrtrr%r4)rArlryrtZall_premise_tokensZall_hypothesis_tokensrrrrB s    zSNLIDataset.__init__cstfdd|DS)Ncs(g|] }tj|jjdqSr)rrrtryrGr@rrrs s z$SNLIDataset._pad..)rrJ)rArJrr@rrz szSNLIDataset._padcCs|j||j|f|j|fSr)rvrwrtrrrrr szSNLIDataset.__getitem__cCs t|jSr)r4rvr@rrrr, szSNLIDataset.__len__)N)rMrNrOrPrBrzrr,rrrrry s  ryrc Csrt}td}t|d}t|d}t||}t|||j}tjj||d|d}tjj||d|d} || |jfS)zDownload the SNLI dataset and return data iterators and vocabulary. Defined in :numref:`sec_natural-language-inference-and-dataset`rlTFr) rrrrxryrtrrgri) rbryrrrfrgr]test_setrrrrrload_data_snli s      r|cs0ttfdd|D}|tjj|ddfS)zwSplit multi-input `X` and `y` into multiple devices. Defined in :numref:`sec_natural-language-inference-attention`csg|]}tjj|ddqSFZ even_splitrrlrmrr rrrs s z,split_batch_multi_inputs..Fr~)r/r6rrlrmrnrr rsplit_batch_multi_inputs s rcCsjtj||td}tj||td}tj||d|dgdd}|dkrZdS|dkrfdSdS) zPredict the logical relationship between the premise and hypothesis. Defined in :numref:`sec_natural-language-inference-attention`r)r+rEr+rrrnrorp)rrJrr rrV)rrtZpremiseZ hypothesisrrrr predict_snli sr)z:https://files.grouplens.org/datasets/movielens/ml-100k.zipZ(cd4dcac4241c8a4ad7badc7ca635da8a69dddb83ml-100kcCs\td}ddddg}tjtj|dd|dd }|jj d }|j j d }|||fS) Nruser_iditem_idrating timestampzu.datarpython)namesenginer) rrrrrrrrrrUr)rrrg num_users num_itemsrrrread_data_ml100k s  rr|rOcs@|dkriig}}}|D]h}|d|d|d|df\} } } } || g| | | | f| |ksz|| d| kr | | | f|| <q td|dD]} |t|| ddd qd d |Dfd d |D} t| } tnFd d t j ddt |d|kD}dd |D}||||} | fS)z3Split the dataset in random mode or seq-aware mode. seq-awarer+r\rrrEcSs|dS)Nrrrrrrr rTz#split_data_ml100k..rLcSsg|]\}}|f|qSrr)rprVrrrrrs sz%split_data_ml100k..csg|]}|kr|qSrr)rprrgrrrs scSsg|]}|dkrdndqS)r+TFrrrrrrs srcSsg|] }| qSrrrrrrrs s) itertuples setdefaultrFrr\r\r]rZ DataFramerr|rr4)rgrr split_mode test_ratioZ train_itemsZ test_itemsZ train_listrHurqrrCrfmaskZneg_maskrrrsplit_data_ml100k s. $   rexplicitc Csggg}}}|dkr&t||fni}|D]}t|ddt|dd} } |dkrlt|dnd} || || || |dkr|| g| q2| || | f<q2||||fS)Nrr+r\rimplicit)rr rrnrFr) rgrrfeedbackZusersr]rZinterrHZ user_index item_indexrrrrload_data_ml100k s "   rrrtcCst\}}}t|||||\}}t||||\} } } } t||||\} }}} tjt| t| t| }tjt| t|t|}tjj|dd|d}tjj||d}||||fS)NTrolloverrfrrbr) rrrrrgrhrrJri)rrrrbrgrrrfrgZtrain_uZtrain_iZtrain_rrZtest_uZtest_iZtest_rr]r{rrrrrsplit_and_load_ml100k$ sR   rc st} tjdd|gddgddgd} t|D]} tdd } } t|D]\}}| g}t|trp|n|g}|D]}| t j ||qzt |dkr|dd n|}|d }t4fd d t|D}fd d t||D}W5QRXdd |D| tdd |Dt |7} ||djd| | |djd|dj| qLt |dkr|||d|}n |||}| |d}| | d||fq.td| d| ddd|dt| d|| ddt|dS)Nrr+rr\rz test RMSErrrrEcsg|] }|qSrrrptrxrrrsI sz'train_recsys_rating..csg|]\}}||qSrr)rprrrrrrsJ scSsg|] }|qSrrrrrrrsK scSsg|] }|qSrasnumpyrrrrrsL sZ inter_mat train loss rz , test RMSE r r!)rr=rrrryr?r.r/rFrrlrmr4rrr6rHrrrUrrrGrr%)rrrrr&rr  evaluatorr9r'rrrrrqr input_datarOZ train_featZ train_labelrrZ test_rmser)rrrtrain_recsys_rating8 s> ""    $rcs&eZdZdfdd ZddZZS)BPRLossNrc s tt|jfddd|dSNr)weight batch_axis)r/rrBrArrr9r:rrrB] szBPRLoss.__init__cCs*||}tjtt|ddd }|S)NrT)Zkeepdims)rrHrrZsigmoid)rArir distancesrrrrr>` szBPRLoss.forward)NrrMrNrOrBr>r@rrr:rr\ srcs(eZdZdfdd Zd ddZZS) HingeLossbRecNrc s tt|jfddd|dSr)r/rrBrr:rrrBf szHingeLossbRec.__init__r+cCs$||}tt| |d}|Sr)rrHr)rArirmarginrrrrrr>j szHingeLossbRec.forward)Nr)r+rrrr:rre srcstfddt|d|D}fddt|D}t|d}t|dkrdd||dd|nd}t||fS)Ncs$g|]\}}|tkr||fqSrrrprval test_matrixrrrsp s zhit_and_auc..cs$g|]\}}|tkr||fqSrrrrrrrsr s r+rrN)ryr4)Z rankedlistrrZhits_kZhits_allraucrrr hit_and_auco s  (rcsiiggf\}}} } tddt|D} t|D]lt| t|t} ggggf\} }fdd| Dfdd| D| tg|dk r| |ddf| tgtj j tj j | dddd}t |D]6\}}fd d|D} |fd dt | Dqd d|D}tt |}t|d d dd|<dd|D|<t||d}| |d| |dq2tt| tt| fS)NcSsg|]}|qSrrrorrrrs{ sz$evaluate_ranking..csg|]}|qSrrFro)item_idsrrrs scsg|]}qSrrr)ruser_idsrrrs sFrircsg|]}tjj|ddqSr}rrNr rrrs scsg|]}t|qSr)r/rrrxrrrs scSsg|]}|D]}|q qSrr)rpZsublistrrrrrs scSs|dSrRr)rrrrr rTz"evaluate_ranking..TrUcSsg|] }|dqSrMr)rprrrrrs srrr+)rrr/rnr\rrJrFrrgrirhryr6r\rr)rZ test_inputseqrrrr Z ranked_listZ ranked_itemshit_raterZ all_itemsZ neg_itemsr:rZtest_data_iterrkrZ item_scorestempr)r rrrrrevaluate_rankingx s<  rr+c  stdd} }}tjdd|gddgddgd}t|D]x}tdd}}t|D]\}}g}|D]}|tj || qjt dfd d t |dd D}fd d t |dd |d fD}fdd t ||D}W5QRXdd |D|t dd |Dt| 7}||djd|||djd|dj| qXt D|d| dkr| ||| ||| \}}||d||fW5QRXq:td|d|dddt|ddt|dt|d|| ddt| dS)Nrrr+z test hit rateztest AUCrrrcsg|] }|qSrrrrxrrrs sz!train_ranking..rEcsg|] }|qSrrrrxrrrs srDcsg|]\}}||qSrr)rprrrrrrs scSsg|]}|jddqS)F)Z retain_graphrrrrrrs scSsg|] }|qSrrrrrrrs srrz, test hit rate z , test AUC r\r r!)rr=rrrryrFrrlrmrrr6rHrr4rrUrrrGZ predict_moderrr%)rrrrr&Z test_seq_iterrrrr rrZ eval_stepr'rrrrrrrqrrrOZp_posZp_negrrrr train_ranking s@  ""   4rzctr.zipZ(e18327c48c8e8e5c23da714dd614e390d369843fZctrc@s&eZdZd ddZddZdd ZdS) CTRDatasetNr"c s|di|_|_|_tdd}|||_|_tj|jtjd|_ t |}|D]}i} | d d} t | |jdkr~qRtddg} d| t| d<t| dg| d<td|jdD]2} || | | d7<| d g| | q| |j|j<|jd|_qRW5QRX|jdkrn|jdkrnfd d |D}d d |D|_d d |D|_|jD] \} } t | d|j | d<qxtdt|j dd|_dS)NrcSsttSr)r rnrrrrr rTz%CTRDataset.__init__..rRrrr+r;r:cs(i|] \}}|fdd|DqS)csh|]\}}|kr|qSrr)rpfeatr min_thresholdrr s z1CTRDataset.__init__...)r])rprqZcntrrrrZ sz'CTRDataset.__init__..cSs$i|]\}}|ddt|DqS)cSsi|]\}}||qSrr)rprZfeat_vrrrrZ sz2CTRDataset.__init__...)ryrprqZ feat_valuesrrrrZ scSsi|]\}}|t|qSrrrrrrrZ srE)r)Z NUM_FEATScountrgr feat_mapperdefaultsrr int64Z field_dimsrrrr4rXrnrrrFr]rJrKroffsets)rAZ data_pathrrrZnum_featZ feat_cntsrrHinstancerrrqfmrrrrB s>    zCTRDataset.__init__cCs|jSr)rr@rrrr, szCTRDataset.__len__cs>tfddtj|dD}|jj|dfS)Ncs0g|](\}}j|d|j|dqSr{)rrr)rprqrOr@rrrs sz*CTRDataset.__getitem__..r:r;)rrJryrgr)rArrrr@rr szCTRDataset.__getitem__)NNrr)rMrNrOrBr,rrrrrr s  rc Cs|jd}tj|f|jd}tj|f|jd}t:||} ||} || } || ||| |d} W5QRX| | |t | S)z=Update discriminator. Defined in :numref:`sec_basic_gan`rrr\) rUronesrr rrrrrrrH) r1Znet_Dnet_GrZ trainer_Drbrr Zreal_Yfake_Xfake_YZloss_Drrrupdate_D s   " rc Csh|jd}tj|f|jd}t ||}||}|||} W5QRX| ||t| S)z9Update generator. Defined in :numref:`sec_basic_gan`rr) rUrrrrrrrrrH) rrrrZ trainer_GrbrrrZloss_Grrrupdate_G s   rz pokemon.zipZ(c065c0e2593b8b161a2d7873e42418bf6a21106cZpokemoncCs|jSr)rrrrrr rTrcCs|jSr)rrrrrr rTcOs |j||Srrr:rr9rrrr rTcOs |j||Sr)rVrrrrr rTcOs |j||Sr)Z as_in_contextrrrrr rTcOs |j||Sr)rHrrrrr! rTcOs |j||Sr)rrrrrr" rTcOs |j||Sr)rrrrrr# rT)r) NNNNNNr'r'r(rN)T)Nrv)N)r)N)r)N)rL)rE)Fr)F)N)r)F)Nrr)rBN)rrT)r\)r\)NN)r)r)rr)T)T)N)rd)r)r|rO)r)rrrOrt)r+)Z USE_MXNETZ USE_PYTORCHZUSE_TENSORFLOWrArZDATA_URLZmxnetrrrrrrrZ mxnet.gluonr r Zmxnet.gluon.data.visionr rZ nn_Modulerqrrrr|rCrrrrCrr ZpandasrrZIPythonr Z matplotlibrrZmatplotlib_inlinermodulesrMrrrr&r7r=rZr[r^rdrmrurrrrrrrrrrrrrrrrr r rrr*r+rKrPrQr[rwrrrrrrrrrrrrrrrrrrrrrrZ SoftmaxCELossrrrrrrrrrrrrr#r,r2r5r9r@rKrRrZrdrfrgrorvr~rrrrrrrrrrrrrrrrgZDatasetrrrrZ VOC_CLASSESrrrrrrrrrrrr r rrrrr*r+r/r3r4r=rArCrIrQrZr[r^r_rcrhrkrxryr|rrrrrrrZLossrrrrrrrrrr!rr r&rQr(sinhr)coshrlinspacerrrJr~rSranddotrTrrXrrrabseyer{rVtorrrrrrrs$      $  !      )     %  ,          !   (               '   #  #    (                            #     $       $  !  (