How to resolve Hadoop warning “WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable“? this is one of the frequently asked questions on Hadoop and Spark community. If you are running Spark on windows with Eclipse or IntelliJ you would also get this warning on the console output.
Question: How to resolve warning “Unable to load native-hadoop library for your platform”
I have installed and running Hadoop on a Unix based server, When I run start-dfs.sh or stop-dfs.sh, I get the following error message.
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
After searching online, I come to know that contents of /native/ directory on hadoop 2.x are different.
I’ve also added these two environment variables in hadoop-env.sh file.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/"
export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"
Solutions
There are multiple solutions to rid of this warning “Unable to load native-hadoop library for your platform“. It all depends on which OS you are running Hadoop on and the version of Hadoop you are using.
First of all, note that this is a safe warning, not an error hence, you can live with the warning as it doesn’t impact the runtime or function of the Hadoop. However, if you are bothered about the warning please flow below steps to rid of the message.
The following solutions will work on CentOs, Ubuntu, macOS, and many more. Based on the OS you use, you may need to adjust the paths and versions.
If you are getting this warning while running Hadoop on windows or by running Spark using Eclipse or IntelliJ, you just need to ignore this warning and there is no solution to get rid of the warning as windows don’t have share libraries.
Solution 1: Hadoop Native Library Version Mismatch
You will get this warning, If you are running Hadoop on 64bit OS and the native Hadoop library $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 was actually compiled on 32 bit.
In order to resolve this warning, download the source code of Hadoop and recompile libhadoop.so.1.0.0 on 64bit system, then replace the 32bit one.
You can check the version of Hadoop library by running below commands
cd $HADOOP_HOME/lib/native
ldd --version
ldd $HADOOP_HOME/lib/native/libhadoop.so.1.0.0
linux-vdso.so.1 (0x00007fff149f3000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcfb2d56000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcfb2b37000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcfb2746000)
/lib64/ld-linux-x86-64.so.2 (0x00007fcfb317c000)
Here, I am using 64 bit Hadoop library, If you already have 64 bit Hadoop library or If you feel this is too much, please skip this section and move on to the next.
If you wanted to continue, follow the below link for the steps of how to re-build the library on Ubuntu.
Solution 2: Update Hadoop Environment variables
You would be getting this warning if you miss setting up Hadoop library environment variables.
Before proceed, make sure you have $HADOOP_HOME with the hadoop installation directory.
Edit the .bashrc file.
vi ~/.bashrc
Added below environment variables, on top of what you already have.
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"
Now reload the environment variables by running source command.
source ~/.bashrc
This should resolve your issue, if not please try below step.
Solution 3: Add Hadoop Native Library to OS Shared Library
Added Hadoop native library to LD_LIBRARY_PATH on .bashrc file and reload the library into the current session using source ~/.bashrc
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
# (OR) If you have hadoop library installed at /usr/lib/
export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native
If this doesn’t resolve, please try next step below.
Solution 4: Copy the Native Libraries
This is just an alternative to the previous step, If your Hadoop is using libraries from $HADOOP_HOME/lib path, you would need to copy the libraries from /library path into the parent directory.
cp $HADOOP_HOME/lib/native/* $HADOOP_HOME/lib/
Solution 5: Upgrade Java/JDK
Check your Java version, If you are using Java 32 bit, update it to Java 64 bit version and update JDK to 8 or newer versions (Use JDK version suggested for Hadoop version you are using).
Hope one of these solutions resolves your Hadoop warning “Unable to load native-hadoop library for your platform”.
If any of the above solutions work for you, please comment on which one worked so that others might get benefits.
Happy Learning !!
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib/native”
Thank you! You save my day!
Thank you. Going down the list, the first one that worked for me was LD_LIBRARY_PATH (to get rid of these warnings with pyarrow).