Spark Unstructured vs semi-structured vs Structured data

  • Post author:
  • Post category:Apache Spark
  • Post last modified:March 21, 2020

Spark RDD natively supports reading text files and later with DataFrame, Spark added different data sources like CSV, JSON, Avro, Parquet and many more. Based on the data source you choose, you may need a third party dependency and Spark can read and write all these files from/to windows(using Uinutils), Linux, HDFS, S3, Azure, GCP, and many more cloud platforms.

Unstructured data

Text file formats are considered unstructured data. In order to process text files use <a href="https://sparkbyexamples.com/spark/spark-read-text-file-rdd-dataframe/">spark.read.text()</a> and <a href="https://sparkbyexamples.com/spark/spark-read-text-file-rdd-dataframe/">spark.read.textFile()</a>

Semi-Structured data

CSV and TSV is considered as Semi-structured data and to process CSV file, we should use <a href="https://sparkbyexamples.com/spark/spark-read-csv-file-into-dataframe/">spark.read.csv()</a>

XML and JSON file format is considered semi-structured data as the data in the file can represent as a string, integer, arrays e.t.c but without explicitly mentioning the data types.

Processing JSON file in spark can be done using  <a href="https://sparkbyexamples.com/spark/spark-read-and-write-json-file/">spark.read.json("path")</a> or <a href="https://sparkbyexamples.com/spark/spark-read-and-write-json-file/">spark.read.format("json").load("path")</a>

Note that Parsing unstructured and semi-structured data to DataFrame and Dataset is very slow.

Structured data

Avro and Parquet file formats are considered structured data as these can maintain the structure/schema of the data along with its data types.

avro() function is not provided in Spark DataFrameReader  hence, we should use DataSource format as “avro” or org.apache.spark.sql.avro and load() is used to read the Avro file. pass HDFS path as an argument to the load function.

DataFrameReader provides parquet() function (spark.read.parquet) to read the parquet files and creates a Spark DataFrame.

NNK

SparkByExamples.com is a Big Data and Spark examples community page, all examples are simple and easy to understand and well tested in our development environment Read more ..

Leave a Reply

You are currently viewing Spark Unstructured vs semi-structured vs Structured data