Spark JSON data source API provides the multiline option to read records from multiple lines. By default, spark considers every record in a JSON file as a fully qualified record in a single line hence, we need to use the multiline option to process JSON from multiple lines.
Using multiline Option – Read JSON multiple lines
In this example, we set multiline option to true to read JSON records from multiple lines into Spark DataFrame. By default, this option is set to false. Let’s consider we have a below JSON file by name “multiline-zipcode.json”. this file is also available at GitHub for reference
[{
"RecordNumber": 2,
"Zipcode": 704,
"ZipCodeType": "STANDARD",
"City": "PASEO COSTA DEL SUR",
"State": "PR"
},
{
"RecordNumber": 10,
"Zipcode": 709,
"ZipCodeType": "STANDARD",
"City": "BDA SAN LUIS",
"State": "PR"
}]
Use spark.read.option("multiline", "true") while reading
//read multiline json file
val multiline_df = spark.read.option("multiline", "true")
.json("src/main/resources/multiline-zipcode.json")
multiline_df.printSchema()
multiline_df.show(false)
multiline_df.printSchema() yields below schema
root
|-- City: string (nullable = true)
|-- RecordNumber: long (nullable = true)
|-- State: string (nullable = true)
|-- ZipCodeType: string (nullable = true)
|-- Zipcode: long (nullable = true)
multiline_df.show(false) statement yields below output
+-------------------+------------+-----+-----------+-------+
|City |RecordNumber|State|ZipCodeType|Zipcode|
+-------------------+------------+-----+-----------+-------+
|PASEO COSTA DEL SUR|2 |PR |STANDARD |704 |
|BDA SAN LUIS |10 |PR |STANDARD |709 |
+-------------------+------------+-----+-----------+-------+
Complete example of reading JSON using multiline option
val spark: SparkSession = SparkSession.builder()
.master("local[3]")
.appName("SparkByExamples.com")
.getOrCreate()
This complete example is available at GitHub.
Conclusion
In this article, you have learned how to read a JSON from multiline and convert it into Spark DataFrame using a Scala example.
Happy Learning !!
Thanks Marcio for your comments.
OMG, thank you for this post, it helps and teaches me a lot!!