Set up Hadoop in pseudo distributed mode
Posted By : Md Qasim Siddiqui | 28-Dec-2014
Following my previous blog on 'Set up Hadoop on single node', This Blog will explain how to set up hadoop in pseudo-distributed mode
-
Configure the core-site.xml,mapred-site.xml and hdfs-site.xml present in /opt/hadoop-2.5.1/etc/hadoop
-
core-site.xml
fs.default.name hdfs://localhost:9000 The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation.
-
Hadoop’s default configurations use hadoop.tmp.dir as the base temporary directory both for the local file system and HDFS
-
Change hadoop.tmp.dir property to /app/hadoop/tmp in core-site.xml like this
hadoop.tmp.dir /app/hadoop/tmp A base for other temporary directories. fs.default.name hdfs://localhost:9000 The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation.
- mapred-site.xml.template
mapred.job.tracker hostname:8021
- hdfs-site.xml
dfs.replication 1 Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
-
Create hadoop tmp directory , sudo mkdir -p /app/hadoop/tmp
-
change ownership permission of tmp in /app/hadoop/tmp to user from root , tmp directory user permission should be same as hadoop directory
-
sudo chown user:user /app/hadoop/tmp
-
Now format the namenode , Type hdfs namenode -format
-
Note: If you forget to set the required ownerships and permissions, you will see a java.io.IOException when you try to format the name node
-
Now start the hadoop services , /opt/hadoop-2.5.1/sbin//start-dfs.sh and /opt/hadoop-2.5.1/sbin//start-yarn.sh
-
The hadoop daemon log output is written to the ${HADOOP_LOG_DIR} directory (defaults to ${HADOOP_HOME}/logs).
-
Type jps, to check all hadoop daemons started successfully
12882 Jps
12320 SecondaryNameNode
12592 NodeManager
12106 DataNode
11978 NameNode
12468 ResourceManager
Congratz Ur Hadoop installed successfully!
Cookies are important to the proper functioning of a site. To improve your experience, we use cookies to remember log-in details and provide secure log-in, collect statistics to optimize site functionality, and deliver content tailored to your interests. Click Agree and Proceed to accept cookies and go directly to the site or click on View Cookie Settings to see detailed descriptions of the types of cookies and choose whether to accept certain cookies while on the site.
About Author
Md Qasim Siddiqui
Qasim is an experienced web app developer with expertise in groovy and grails,Hadoop , Hive, Mahout, AngularJS and Spring frameworks. He likes to listen music in idle time and plays counter strike.