Spark | Hadoop – Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

When you run the Spark program on windows OS, you often get the exception “Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z”

Full-stack trace of the error


Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

Solution: In order to run the Spark program on windows, you would need Hadoop winutils.exe file as windows don’t support HDFS and winutils provides a wrapper.

If you don’t have winutils.exe installed, please download the wintils.exe and hadoop.dll files from https://github.com/steveloughran/winutils (select the Hadoop version you are using as winutils are specific to Hadoop versions)

Copy the downloaded files to a folder on your system, for example, let’s say you have copied these files to c:/hadoop/bin, set the below environment variables.


set HADOOP_HOME=c:/hadoop
set PATH=%PATH%;%HADOOP_HOME%/bin;

Close and reload the command line or terminal to initialize these variables. Sometimes you may also need to put hadoop.dll file into the C:/Windows/System32 folder.

Now run your spark program and issue “Windows.access0(Ljava/lang/String;I)Z” should disappear.

However, if it still doesn’t work, try to restart your system as some of the above settings would get the effect with the restart.

Happy Learning !!

Tags:

Naveen Nelamali

Naveen Nelamali (NNK) is a Data Engineer with 20+ years of experience in transforming data into actionable insights. Over the years, He has honed his expertise in designing, implementing, and maintaining data pipelines with frameworks like Apache Spark, PySpark, Pandas, R, Hive and Machine Learning. Naveen journey in the field of data engineering has been a continuous learning, innovation, and a strong commitment to data integrity. In this blog, he shares his experiences with the data as he come across. Follow Naveen @ LinkedIn and Medium

This Post Has 12 Comments

  1. Anonymous August 28, 2022

    A life saver

  2. ABHISHEK RAJBHANU May 26, 2022

    For me winutil.exe + hadoop.dll , Thank you

  3. NNK May 4, 2022

    Technically you don’t need Hadoop to run PySpark on windows. you need winutils dll/exe files in the right path.

  4. Corey May 4, 2022

    Does this mean hadoop needs to installed on windows? Or, as long as i have installed pyspark and the dll/exe I would be good

  5. Aditya April 26, 2022

    This solution worked fine.
    Thank you!

  6. Wing October 12, 2021

    Thanks!! Previously I just have the winutils.exe downloaded and was still getting the error. Adding the hadoop.dll helps :)

  7. Raghu June 7, 2021

    Thank you. It worked for me on Windows.

  8. NNK February 21, 2021

    Glad it worked for you. Thanks for sharing.

  9. Anonymous February 21, 2021

    copying dll file to Windows\System32 folder helped me!

  10. Elena December 10, 2020

    Thank you!

  11. NNK October 25, 2020

    Thank you.

  12. diassy_devops October 25, 2020

    merci beaucoup la methode fonctionne bien

Comments are closed.