Spark RDD natively supports reading text files and later with DataFrame, Spark added different data sources like CSV, JSON, Avro, Parquet and many more. Based on the data source you choose, you may need a third party dependency and Spark can read and write all these files from/to windows(using Uinutils), Linux, HDFS, S3, Azure, GCP, and many more cloud platforms.
Text file formats are considered unstructured data. In order to process text files use
<a href="https://sparkbyexamples.com/spark/spark-read-text-file-rdd-dataframe/">spark.read.text()</a> and
TSV is considered as Semi-structured data and to process CSV file, we should use
JSON file format is considered semi-structured data as the data in the file can represent as a string, integer, arrays e.t.c but without explicitly mentioning the data types.
Processing JSON file in spark can be done using
<a href="https://sparkbyexamples.com/spark/spark-read-and-write-json-file/">spark.read.json("path")</a> or
Note that Parsing unstructured and semi-structured data to DataFrame and Dataset is very slow.
Parquet file formats are considered structured data as these can maintain the structure/schema of the data along with its data types.
avro() function is not provided in Spark
DataFrameReader hence, we should use DataSource format as “avro” or
load() is used to read the Avro file. pass HDFS path as an argument to the load function.
DataFrameReader provides parquet() function (spark.read.parquet) to read the parquet files and creates a Spark DataFrame.