Install Hadoop on Mac (M1, M2) with Apple Silicon
- Install
brew
(Homebrew) -
Install
hadoop
brew install hadoop
-
cd
into hadoop configuration filescd /opt/homebrew/Cellar/hadoop/3.3.4/libexec/etc/hadoop
-
Update
hadoop-env.sh
confvim hadoop-env.sh
Find
export JAVA_HOME
line and uncomment it and put your$JAVA_HOME
path in there you can also find it using/usr/libexec/java_home
-
Update
core-site.xml
confvim core-site.xml
Add configuration
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>
-
Update
hdfs-site.xml
vim hdfs-site.xml
Add configuration
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
-
Update
mapred-site.xml
vim mapred-site.xml
Add configuration
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.application.classpath</name> <value> $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* </value> </property> </configuration>
-
Update
yarn-site.xml
vim yarn-site.xml
Add configuration
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME </value> </property> </configuration>
- Enable remote login from “Sharing” in system preferences and allow Administrators from “Only these users:” radio button
-
Run
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-
Run
stop-all.sh hadoop namenode -format start-all.sh
-
Run
jps
to verify all the installation16641 NameNode 16882 SecondaryNameNode 17170 NodeManager 16741 DataNode 17279 Jps 17071 ResourceManager
- Grab popcorns and work with Hadoop!