PyTorch Machine learning framework
代写web | report | 代写Network | network | scheme | Machine learning代写 | oop代做 | Objective | Python | assignment – 这是利用Python进行训练的代写, 对Python的流程进行训练解析, 涵盖了web/report/Network/network/scheme/Machine learning/oop/Objective/Python/IT等程序代做方面, 该题目是值得借鉴的assignment代写的题目
1 Objective
The Objective of this assignment is to practise the use of thePyTorch Machine learning framework through building two deep learning models to solve two interesting and realistic tasks around the common theme of movies. Throughout the assignment, students will be guided to develop the models step by step and study the effects of different model settings.
2 Major Tasks
The assignment consists of two parts and a written report:
Part 1:Build an end-to-end multi-label convolutional neural network (CNN) model to predict the genres of movies from their posters.
Part 2:Build an end-to-end recurrent neural network (RNN) model to predict the sentiments of movie reviews.
Written Report: report the results and answer some questions.
The two parts will be elaborated in Sections 4 and 5, respectively. Note that [Qn] refers to a specific question (thenth question) that you need to answer in the written report.
3 Setup
- Make sure that the librariestorch,torchvision,torchsummary,torchtext,matplotlib andsklearnhave been installed on your machine.
- For this assignment, IT suffices to use onlyPyTorchwith Python 3.7or above. Other machine learning frameworks includingTensorFlowandKerasare not accepted.
- You may read the CSSystem GPU user guide and use the GPU servers provided by the Department of Computer Science and Engineering or the Colaboratory, or called Colab, provided by Google (https://colab.research.google.com/).
- The datasets for this assignment are available as a ZIP file (./pa2data.zip). After unzipping the file, you will find the datasets for two parts in./pa2data/part1dataand ./pa2data/part2data, respectively.
4 Part 1: Movie Genre Prediction
For movie lovers, a movie poster often gives the first impression about the movie content and its genre. In this task, your goal is to predict the genres of movies from their posters using a multi-labelCNN model.
4.1 Multi-label Classification
From the lectures and tutorials, you have learned different multi-class classification models which predict exactlyone label per sample. However, in movie genre prediction, one movie can belong tomultiplegenres. Problems of this type are referred to as multi-label classification problems.
You can modify a single-label model to handle multiple labels by changing the target represen- tation and the prediction criterion:
- Represent the multi-label target as multiple binary labels
- During the testing phase, output all the classes that have probability higher than a pre- defined threshold.
Table 1: Representation of multi-label targets
id Class A Class B Class C
x 1 0 0
y 0 1 1
z 1 1 1
Table 2: Prediction (bold) with= 0.
id Class A Class B Class C
x 0.71 0.11 0.
y 0.24 0.62 0.
z 0.82 0.77 0.
4.2 Dataset
The movie poster dataset can be found in the/part1datadirectory. All csv files contain animdbIdcolumn and seven genre columns: Action,Adventure,Animation,Comedy, Drama,Horror, andRomance. All the movie posters are located in the/imagesdirectory named by the moviesimdbId.
The filestrain.csvandtest.csvcontain labeled data, where a 1 in a genre column indicates the presence of that genre. On the other hand, the filesubmission.csvcontains unlabelled data. You have to predict the movie genres insubmission.csvand include the predictions as part of your assignment submission.
4.3 Tasks
This section will guide you to build the CNN model.
Table 3: Description of the data files for Part 1
File # of instances Has label? Purpose
train.csv 10,365 yes training
test.csv 2,000 yes testing
submission.csv 591 no prediction
4.3.1 Dataset and Dataloader
The first step is to build a customDatasetclass. ADatasetobject loads the poster images in the/imagesdirectory using theimdbIdfrom the csv files. You need toapply at least one image transformation operationto process the input images.
In this part, you are required to useholdout validation for hyperparameter tuning. You are suggested to split the data fromtrain.csvinto a training set and a validation set in a 80% : 20% ratio and create theDataloaderobjects.
4.3.2 CNN Model Design
A CNN model takes an image as input and produces class probabilities (or scores) as output. Here, you need to build a CNN model from scratch with the following requirements:^1
- The model should useat least one but less than 10Conv2dlayers.
- The model should useat least onebatchnorm and/or maxpool and/or dropout layer(s).
- The model should haveless than 1,000,000trainable parameters.
- You shouldnotuse any pre-trained or built-in model.^2
[Q1] Print the model architecture usingtorchsummary.summaryand include it in the written report.
4.3.3 Training and Validation
Loss minimization: You can treat a multi-label model as multiple binary classifiers, each of which predicts the presence of one class. You can use theBinary Cross Entropy Lossor Binary Cross Entropy Loss with Logitsprovided byPyTorchas the loss function depend- ing on your model output.^3 You are free to use any optimizer.^4
Training:During training, you are required to plot the training and validation losses over time usingTensorboardormatplotlib.^5 You are advised to adjust your model architecture if the training loss does not drop after 10 epochs. By now, you should have a working model with a decreasing loss.
[Q2] Paste the screenshots of the plots and report the final training and validation losses ob- tained.
(^1) You should follow these requirements throughout Part 1. (^2) You can look at some popular CNN architectures such as VGGNet and ResNet as reference. (^3) Optional to try other multi-label losses. (^4) Optional to try some optimizing techniques such as learning rate decay, special initialization, etc. (^5) Optional to monitor other metrics, such as F1 score, for a better understanding of the learning performance.
Hyperparameter Tuning:A lot of things can be modified to improve the performance of the CNN model, such as the model architecture, optimizer, batch size, learning rate, loss, or even data augmentation. Based on your observation in Q2, make at least one change to improve your model.^6 For example, you can change the learning rate, change the batch size, add more convolutional layers, or augment the data, etc.
[Q3] Discuss the change(s) and improvement and report the final hyperparameter setting. You may discuss the results in terms of the running time, training loss, validation loss, convergence rate, etc. You are expected to use graphs, tables or other visual means to assist your analy- sis.
4.3.4 Testing
You are required to compute the classification report fromsklearn.metricsby training the CNN model with the whole training set and testing it againsttest.csvusing the optimal hyperparameters found in the previous holdout validation. You may want to train the model with more epochs for a better result. Please be reminded that a multi-label model selects the classes with probability larger than a predefined thresholdas the prediction. Repeat the evaluation by choosing different values of and observe how it affects the precision and recall.
[Q4] Paste the screenshots of the classification reports. Describe how varying affects the precision and recall. Explain the reason behind.
[Q5] Given that the cost of misclassification is higher than that of missing the true label(s), how should you set the threshold and why?
Note: Multi-label classification is more difficult than single-label classification because the in- put images and output labels are more complex. So, do not worry if you cannot achieve high precision, high recall, or high F1 score. It is more important that you demonstrate your under- standing in building and tuning a CNN model from end to end. Beyond the assignment, you may borrow ideas from some popular CNN models like ResNet or VGGNet and build a deeper CNN model for better results.
4.3.5 Prediction
Finally, you have built your own movie genre classification model! The last step is to predict the movie genres insubmission.csvand save the predictions as a csv file. Your predictions should be aimed to produce an optimal F1 score.
[P1] Submit your predictions as a csv file. The format should be the same as that oftrain.csv: oneimdbIdcolumn and seven genre columns in the same order, where 1 in the data rows indicates the presence of a genre. The order of theimdbIdcolumn should be the same as that insubmission.csv.
(^6) You may apply grid search or random search. However, try to limit the search space to avoid having a long running time.
5 Part 2: Movie Review Sentiment Prediction
After watching a movie, some people write up their reviews on the web. These reviews can be roughly categorized into positive reviews and negative reviews. Since text is sequential in nature, it is common to use RNNs for textual data. In this part, you are required to build a binary RNN classifier to predict the sentiments of movie reviews.
5.1 Binary Text Sentiment Classification
From a tutorial, you have learned to use an RNN model to predict stock price. For text classifi- cation, the workflow is also similar. The RNN model takes a sentence (as a sequence of words) as input and models the sequential relationships between the words to predict a sentiment score.^7
5.2 Dataset
The movie review dataset can be found in the directory/part2data. For all csv files, each row represents one movie review. Fortrain.csv,validation.csvandtest.csv, there are one labelcolumn and onetextcolumn. The label column indicates the sentiments of the reviews (0: Negative, 1: Positive); the text column contains the textual movie reviews.
Forsubmission.csv, all the labels are set to 3 , meaningUnknown. You are required to predict and submit the predicted labels of the reviews insubmission.csv.
Table 4: Description of the data files for Part 2
File # of instances Has label? Purpose
train.csv 40,000 yes training
validation.csv 5,000 yes validation
test.csv 3,000 yes testing
submission.csv 2,000 no prediction
5.3 Tasks
This section will guide you to build the RNN model. Please create a newJupyternotebook for this part.
5.3.1 Dataset and Dataloader
The first step is to build a customTabularDatasetclass. ATabularDatasetobject loads the data fromtrain.csv,validation.csvandtest.csvwith properly defined fields for label data and text data.
For this part, you do not need to split the training set further. You should train the model on the training set and evaluate it on the validation set provided. (Hint: the vocabulary of training, testing and predicting stages should only be built on the training data provided.)
(^7) You may refer to the tutorial notes on some related concepts and techniques for dealing with textual data, such as tokenization, vocabulary, word embedding, etc.
5.3.2 RNN Baseline Model
An RNN model takes a sequence of words as input and produces class probabilities (or scores) as output. Here, you need to build a baseline RNN model with the following settings:
Table 5: Configuration of the baseline model
Parameter Value Remarks
# embedding layers 1 embedding dimension = 64
# RNN layers 1
RNN cell =nn.RNN
hidden dimension = 128
# dropout layers 1 p= 0. 5
# fully connected layers 1
Batch size 32
Optimizer Adam
[Q6] Print the model architecture and the number of trainable parameters for the model and include it in the written report. (Hint: sincetorchsummary.summarydoes not support multi- input RNN model directly, you may use other functions introduced in the tutorial notes.)
Loss Minimization: Since this is a binary classification task, you may useBinary Cross Entropy LossorCross Entropy Loss. (Hint: there are slight differences for the input of the two losses.) provided byPyTorch. For optimization, you are advised to start with the default setting of the Adam optimizer.
Training: During training, you are required to plot the training and validation losses in one plot usingmatplotlib.^8 The model should converge within 10 epochs. It is useful for you to save the training and validation loss profiles as well as the test accuracy for later parts.
Testing: You are required to compute the accuracy of the baseline model against the test set test.csv. You may combine the training and validation sets to train the model. We will name this model thebaseline modelfor the rest of the assignment.
[Q7] Paste the screenshot of the plot and report the test accuracy.
5.3.3 Empirical Study
Different RNN Settings
In this section, you need to build four models in addition to the baseline model. These four models are variants of the baseline model and are used to compare the performance of different RNN settings. You should only change the settings indicated in Table 6 and keep the other parameters the same as the baseline model.
Model Description: Models 1 and 2 replace the baseline vanilla RNN cell with GRU and LSTM cells. Model 3 uses bidirectional LSTM. Bidirectional here means that the sequential relationship of the sequences is modeled in both forward and backward directions. This can be set in the parameters ofnn.LSTM. Model 4 examines the effect of an embedding layer. You need to remove thenn.Embeddinglayer from the baseline model and connect the input words directly to the LSTM cells.
(^8) Optional to monitor other metrics, such as F1 score, for a better understanding of the learning performance.
Table 6: Configurations of models 1-
RNN cell Bidirectional? Embedding layer?
Model 1 nn.GRU no yes
Model 2 nn.LSTM no yes
Model 3 nn.LSTM yes yes
Model 4 nn.LSTM no no
Training and Testing: Same as the baseline model, you are required to keep track of the training and validation losses of all four models. You also need to report the test accuracies of the four models as what you did for the baseline model. It will be very useful for you tosave the trained weights of the models, especially model 2, for the later task.
[Q8] Plot the training and validation losses over time of thebaseline model,model 1andmodel 2 in one plot. Briefly explain your observation and suggest some possible reason(s).
[Q9] Plot the training and validation losses over time ofmodel 2 andmodel 3 in one plot. Briefly explain the effect of adding the backward direction in bidirectional LSTM compared to the standard LSTM.
[Q10] Plot the training and validation losses over time ofmodel 2 andmodel 4 in one plot. What is the effect of removing the embedding layer?
[Q11] Rank the performance of thebaseline model,model 1 to model 4according to their test accuracy.
(Hint: you can discuss the training and validation losses, convergence rate, and the gap between losses in your analysis.)
Word Embedding:
From the above analysis, you may notice that the embedding layer plays an important role in text processing. When you train the RNN model, you also train the embedding layer. Such embedding layer is called aword embeddingin the context of natural language processing. It embeds discrete words into numerical vectors, which carry the meanings of the words. In this subsection, you will investigate the embedding trained frommodel 2.
Load the trained model 2 and compute the L2 distance between the word embeddings of the words:
- good and happy
- boring and bad
- good and boring
[Q12] Report the L2 distances of the three word pairs. What do you observe?
5.3.4 Prediction
In the prediction stage, choose the model you ranked highest in [Q11] and predict the sentiment labels of the movie reviews insubmission.csv. Save the predictions as a csv file.
[P2] Submit your prediction as a csv file. The format should be the same astrain.csv: one label column and one text column in the same order. 0 and 1 in the label rows indicate the negative and positive sentiments of a review, respectively.
6 Written Report
Answer [Q1] to [Q12] in one single written report.
7 Some Programming Tips
As is always the case, good programming practices should be applied when coding your program. Below are some common ones but they are by no means complete:
- Using a small subset of data to test the code
- Using checkpoints to save partially trained models
- Using functions to structure program clearly
- Using meaningful variable and function names to improve readability
- Using consistent styles
- Including concise but informative comments
8 Assignment Submission
Assignment submission should only be done electronically using the Course Assignment Sub- mission System (CASS):
https://cssystem.cse.ust.hk/UGuides/cass/student.html
There should be five files in your submission with the following naming convention required:
- Reportfor part 1 and part 2 (with filenamereport.pdf): in PDF format.
- Predictionfor part 1 (with filenamep1.csv): in csv format.
- Predictionfor part 2 (with filenamep2.csv): in csv format.
- Part 1 source code and a README file(with filenamecode1): com- pressed into a single ZIP or RAR file.
- Part 2 source code and a README file(with filenamecode2): com- pressed into a single ZIP or RAR file.
Note: The source code files should contain all necessary code for running your program as well as a brief user guide for the TA to run the programs easily to verify your results, all compressed into a single ZIP or RAR file. The data should not be submitted to keep the file size small. When multiple versions with the same filename are submitted, only the latest version according to the timestamp will be used for grading. Files not adhering to the naming convention above will be ignored.
9 Grading Scheme
This programming assignment will be counted towards 20% of your final course grade. The maximum scores for different tasks are as follows:
Table 7: [C]: code, [Q]: written report, [P]: prediction
Grading scheme Code (60) Report (32) Prediction (8)
PART 1: Movie Genre Prediction (30) (16) (4)
Dataset and Dataloader
- [C1] Dataset Class 4
- [C2] Image Transformation 2
- [C3] Training-Validation Split and Dataloader 3 CNN Model Design
- [C4] CNN Model + [Q1] 4 + Training and Validation
- [C5] Loss Function 2
- [C6] Optimizer 1
- [C7] Training L oop 4
- [C8] Validation Loop 4
- [Q2] Loss Curve 4
- [Q3] Hyperparameter Tunning 4 Testing
- [C9] Test Evaluation + [Q4] 4 +
- [Q5] Misclassification Cost and Label-missing Cost 2 Prediction
- [C10] Prediction on Submission Images + [P1] 2 + PART 2: Movie Review Sentiment Prediction (30) (16) (4) Dataset and Dataloader
- [C11] Dataset Class 4
- [C12] Vocabulary 2 RNN Model
- [C13] Baseline Model + [Q6] 4 +
- [C14] Model 1 2
- [C15] Model 2 2
- [C16] Model 3 2
- [C17] Model 4 2 Training and Testing
- [C18] Loss Function 2
- [C19] Training Loop 2
- [C20] Validation Loop 2
- [Q7] Baseline Loss Curves and Test Accuracy 2 + Empirical Study
- [Q8] Comparison of Different Recurrent Units 3
- [Q9] Comparison of Bidirectional and Standard LSTM 2
- [Q10] Comparison of Use of Embedding Layer 2
- [Q11] Ranking of Models 2
- [Q12] Word Embedding 2 + Prediction
- [C21] Prediction on Submission Movie Reviews + [P2] 2 +
Late submission will be accepted but with penalty. The late penalty is deduction of one point (out of a maximum of 100 points) for every minute late after 11:59pm. Being late for a fraction of
a minute is considered a full minute. For example, two points will be deducted if the submission time is 00:00:34.
10 Academic Integrity
Please read carefully the relevant web pages linked from the course website.
While you may discuss with your classmates on general ideas about the assignment, your sub- mission should be based on your own independent effort. In case you seek help from any person or reference source, you should state it clearly in your submission. Failure to do so is considered plagiarism which will lead to appropriate disciplinary actions.