Install Hadoop 2.8.2 on Mac OSX 10.13 High Sierra
0
views
Installation
Install hadoop via homebrew is easy!
- $ brew search hadoop
- $ brew install hadoop
The installation log below :
==> Caveats
In Hadoop's config file:
/usr/local/opt/hadoop/libexec/etc/hadoop/hadoop-env.sh,
/usr/local/opt/hadoop/libexec/etc/hadoop/mapred-env.sh and
/usr/local/opt/hadoop/libexec/etc/hadoop/yarn-env.sh
$JAVA_HOME has been set to be the output of:
/usr/libexec/java_home
==> Summary
🍺 /usr/local/Cellar/hadoop/2.8.2: 21,056 files, 618MB, built in 2 minutes 9 seconds
Get hadoop running
Setup properties
$ cd /usr/local/Cellar/hadoop/2.8.2/libexec/etc/hadoop/
hadoop-env.sh
Change $HADOOP_OPTS by adding additional properties : HADOOP_OPTS =
When starting Hadoop using the public distributed version
___________________ | /usr/local/Cellar/hadoop/2.8.2/libexec/etc/hadoop @ gogo (qliang)
$ => hstart
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
18/01/02 23:48:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-qliang-namenode-gogo.out
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-qliang-datanode-gogo.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:1CrU8INzO6C+f8rcQ4HGpfykSXdTLahqHBnTPt/mMYA.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-qliang-secondarynamenode-gogo.out
18/01/02 23:48:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-qliang-resourcemanager-gogo.out
localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-qliang-nodemanager-gogo.out
___________________ | /usr/local/Cellar/hadoop/2.8.2/libexec/etc/hadoop @ gogo (qliang)
$ => jps
96672 Jps
96403 SecondaryNameNode
96293 DataNode
96614 NodeManager
96522 ResourceManager
96207 NameNode
You may notice warning that “Unable to load native-hadoop library for your platform… using builtin-java classes where applicable”. This is because some components of hadoop are provided with both Java and native implementation. There can be performance gain, like the compression algorithm, with the native implementation which however need to be build on your target platform. I am going to log the steps for building a native version(Mac OSX 10.13 in my case) in the following version.
Build Hadoop natively
Prerequisite
- cmake (version >= 3.1.2 when building hadoop version >= 2.7)
- openssl (>=1.0.2)
- zlib
- protobuf (2.5.0)
- maven
Errors encountered while building Hadoop natively
If you had below error while building, you should check you cmake version.
The ' characters around the executable and arguments are
not part of the command.
[exec] -- Configuring incomplete, errors occurred!
[exec] See also "/Users/qliang/dev/distributed-system/hadoop/hadoop-tools/hadoop-pipes/targCMake Error at /usr/local/share/cmake-3.0/Modules/FindPackageHandleStandardArgs.cmake:136 (message):
[exec] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
[exec] Call Stack (most recent call first):
[exec] /usr/local/share/cmake-3.0/Modules/FindPackageHandleStandardArgs.cmake:343 (_FPHSA_FAILURE_MESSAGE)
[exec] /usr/local/share/cmake-3.0/Modules/FindOpenSSL.cmake:328 (find_package_handle_standard_args)
[exec] CMakeLists.txt:24 (find_package)
[exec]
[exec]
[exec] et/native/CMakeFiles/CMakeOutput.log".