Read more about the article Spark from_avro() and to_avro() usage
Photo by Markus Spiske on Unsplash

Spark from_avro() and to_avro() usage

In Spark, avro-module is an external module and needed to add this module when processing Avro file and this avro-module provides function to_avro() to encode DataFrame column value to Avro binary format,…


Using Avro Data Files From Spark SQL 2.3.x or earlier

On this post we will see several examples or usages of accessing Spark Avro file format using Spark 2.3 or earlier. Spark SQL supports loading and saving DataFrames from and to a Avro data files by using spark-avro library. spark-avro originally developed by databricks as a open source library which supports reading and writing data in Avro file format.