Q1: Which are the three modes in which Hadoop can be run?
The three modes in which Hadoop can be run are:
1. standalone (local) mode
2. Pseudo-distributed mode
3. Fully distributed mode
Q2: What are the features of Stand alone (local) mode?
In stand-alone mode there are no daemons, everything runs on a single JVM. It has no DFS and utilizes the local file system. Stand-alone mode is suitable only for running MapReduce programs during development. It is one of the most least used environments.
Q3: What are the features of Pseudo mode?
Pseudo mode is used both for development and in the QA environment. In the Pseudo mode all the daemons run on the same machine.
Q4: What are the features of Fully Distributed mode?
Fully Distributed mode is used in the production environment, where we have 'n' number of machines forming a Hadoop cluster. Hadoop daemons run on a cluster of machines.
There is one host onto which Namenode is running other hosts on which datanodes are running and then there are machines on which task tracker is running. We have separate masters and separate slaves in this distribution.
Q5: In which directory Hadoop is installed?
Hadoop can be installed in any directory, HADOOP_HOME environment variable must point to the installable directory.
Q6: What are the port numbers of Namenode, job tracker and task tracker?
The port number for Namenode is '70', for job tracker is '30' and for task tracker is '60'.
Q7: What are the Hadoop configuration files at present?
There are 3 configuration files in Hadoop:
1. core-site.xml
2. hdfs-site.xml
3. mapred-site.xml
These files are located in the conf/ subdirectory.
Q8: What is a spill factor with respect to the RAM?
Spill factor is the size after which your files move to the temp file. Hadoop-temp directory is used for this.
Q9: Is fs.mapr.working.dir a single directory?
Yes, fs.mapr.working.dir it is just one directory.
Q10: Which are the three main hdfs-site.xml properties?
The three main hdfs-site.xml properties are:
1. dfs.name.dir which gives you the location on which metadata will be stored and where DFS is located – on disk or onto the remote.
2. dfs.data.dir which gives you the location where the data is going to be stored.
3. fs.checkpoint.dir which is for secondary Namenode.
Q11: What happens if you get a 'connection refused java exception' when you type hadoopfsck?
It could mean that the Namenode is not working on your VM.
Q12: How can I restart Namenode?
1. Click on stop-all.sh and then click on start-all.sh OR
2. Write sudo hdfs (press enter), su-hdfs (press enter), /etc/init.d/ha (press enter) and then /etc/init.d/hadoop-0.20-namenode start (press enter).
Q13: How can we check whether Namenode is working or not?
To check whether Namenode is working or not, use the command /etc/init.d/hadoop-0.20-namenode status or as simple as jps.
Q14: What does the command mapred.job.tracker do?
The command mapred.job.tracker lists out which of your nodes is acting as a job tracker.
Q15: What does /etc /init.d do?
/etc /init.d specifies where daemons (services) are placed or to see the status of these daemons. It is very LINUX specific, and nothing to do with Hadoop.
Q16: How can we look for the Namenode in the browser?
If you have to look for Namenode in the browser, you don't have to give localhost:8021, the port number to look for Namenode in the brower is 50070.
Q17: Which files are used by the startup and shutdown commands?
Slaves and Masters are used by the startup and the shutdown commands.
Q18: What do slaves consist of?
Slaves consist of a list of hosts, one per line; datanode and task tracker servers.
Q19: What do masters consist of?
Masters contain a list of hosts, one per line, primary namenode and secondary namenode servers.
Q20: What does hadoop-env.sh do?
hadoop-env.sh provides the environment for Hadoop to run. JAVA_HOME is set over here.
Q21: Can we have multiple entries in the master files?
Yes, we can have multiple entries in the Master files.
Q22:Where is hadoop-env.sh file present?
hadoop-env.sh file is present in the conf location.
Q23: In Hadoop_PID_DIR, what does PID stands for?
PID stands for ‘Process ID’.
Q24: What does /var/hadoop/pids do?
It stores the PID.
Q25: What does hadoop-metrics.properties file do?
hadoop-metrics.properties is used for 'Reporting' purposes. It controls the reporting for Hadoop. The default status is 'not to report'.
Q26: What are the network requirements for Hadoop?
The Hadoop core uses Shell (SSH) to launch the server processes on the slave nodes. It requires password-less SSH connection between the master and all the slaves and the secondary machines.
Q27: Why do we need a password-less SSH in Fully Distributed environment?
We need a password-less SSH in a Fully-Distributed environment because when the cluster is LIVE and running in Fully Distributed environment, the communication is too frequent. The job tracker should be able to send a task to task tracker quickly.
Q28: On which port does SSH work?
SSH works on Port No. 22, though it can be configured. 22 is the default Port number.
Q29: What happens to job tracker when Namenode is down?
When Namenode is down, your cluster is OFF, this is because Namenode is the single point of failure in HDFS.
Q30: What happens to a Namenode, when job tracker is down?
When a job tracker is down, it will not be functional but Namenode will be present. So, cluster is accessible if Namenode is working, even if the job tracker is not working.
Q31: What is formatting of the DFS?
Just like we do for Windows, DFS is formatted for proper structuring. It is not usually done as it formats the Namenode too.
Q32: Does the HDFS client decide the input split or Namenode?
No, the Client does not decide. It is already specified in one of the configurations through which input split is already configured.