Spark Unstructured vs semi-structured vs Structured data

Spark RDD natively supports reading text files and later with DataFrame, Spark added different data sources like CSV, JSON, Avro, Parquet and many more. Based on the data source you choose, you may need a third party dependency and Spark can read and write all these files from/to windows(using Uinutils), Linux, HDFS, S3, Azure, GCP, and many more cloud platforms.

Unstructured data

Text file formats are considered unstructured data. In order to process text files use spark.read.text() and spark.read.textFile()

Semi-Structured data

CSV and TSV is considered as Semi-structured data and to process CSV file, we should use spark.read.csv()

XML and JSON file format is considered semi-structured data as the data in the file can represent as a string, integer, arrays e.t.c but without explicitly mentioning the data types.

Processing JSON file in spark can be done using  spark.read.json("path") or spark.read.format("json").load("path")

Note that Parsing unstructured and semi-structured data to DataFrame and Dataset is very slow.

Structured data

Avro and Parquet file formats are considered structured data as these can maintain the structure/schema of the data along with its data types.

avro() function is not provided in Spark DataFrameReader  hence, we should use DataSource format as “avro” or org.apache.spark.sql.avro and load() is used to read the Avro file. pass HDFS path as an argument to the load function.

DataFrameReader provides parquet() function (spark.read.parquet) to read the parquet files and creates a Spark DataFrame.

NNK

SparkByExamples.com is a BigData and Spark examples community page, all examples are simple and easy to understand and well tested in our development environment using Scala and Maven.

Leave a Reply