COMP4057 Distributed and Cloud Computing
java代写 | 分布式 | project代写 | 代做hadoop – 这是利用java进行训练的代写, 对java的流程进行训练解析, 是比较典型的java/oop/hadoop/分布式等代写方向, 该题目是值得借鉴的分布式方面的代写的题目
lab 1.2 Introduction to Programming on Clus
ter Using IntelliJ
Introduction
The purpose of this document is to setup simple environments (IntelliJ) to write programs that can run on the big-data cluster or on your PC. Note that although you use IntelliJ (IDE, integrated development environment) to build your project, this tutorial still requires you to log in one of the machines in the cluster to submit a job.
Downloading IntelliJ Ultimate
Before downloading IntelliJ, you need to apply for a free student pack through the following
URL: https://www.jetbrains.com/buy/classroom/?fromMenu. You should use your HKBU
email address. After finishing the application form, you will receive an email from JetBrains
team, and remember to confirm the request.
Click it
Then you may download theComm unity version of IntelliJ for your computer through the following URL:https://www.jetbrains.com/idea/download/ You may downloadUltimate version for a trial.
Configuring IntelliJ [Utimate Version only]
Configuring the deployment path
- After installing IntelliJ, launch it and create a new java project. Then, click the menu Tools -> Deployment -> Configuration (to setup a connection with your server)
- In the Deployment dialog window, click +and choose SFTP. Then, type research server for the name. Make sure you have already setup port forwarding if you use this at home.
- In the Connection tab, fill the information as follows:
Host csrXX.comp.hkbu.edu.hk
(where XX is a number in the range from 5 0 to 5 4 )
Port 22
User name and
Password
Your username and password for signing in the research
machines
- In the Mapping tab, map the local path to a path in your research machine, and then chooseok.
- Then, enable automatic upload by clicking the menu Tools -> Deployment -> Automatic upload.
Remark: automatic upload can be useful only if you connect to the HKBUs network
Adding Hadoop
Next, we import the required packages from the Had oop library to IntelliJ. After completing the following steps, IntelliJ can provide suggestions for your coding.
- Download the hadoop library from Moodle (hadoop.rar).
- Extract the files from the compressed file.
- Launch IntelliJ and open the project created previously. Then, click the menuFile -> Project Structure. In the Project Structure dialog window, clickModules -> Dependencies -> + -> JARs or directories…
- In the pop-up window, locate and expand the hadoop directory, and select all its subdirectories. Then, clickOKto attach these files to your project.
Then, when you are programming and using the packages of Hadoop, IntelliJ will give
you suggestions.
- Connect to the node of the cluster. If you are connecting to the campus Intranet, you can use IntelliJ [Ultimate version only] to connect to the cluster directly. Open IntelliJ, click the menuTools -> Start SSH session and select the server you create before, you will see the session start at the bottom of the window.
- After coding, click the menuFile -> Project Structure.
- In the pop-up window, clickArtifacts -> + -> JAR -> From modules with dependencies.
- Fill in the output directory, the .jar file will be generated in this path then.
- Click the menuBuild -> Build Artifacts. If your program is correct, a jar file will be built in the path you fill in before.
10.After you finished the compilation and packaging, you should include the package in the class path in order to let Hadoop to use it:
$ export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/comp/$USER/src/*