PySpark Read and Write Parquet File

Pyspark SQL provides methods to read Parquet file into DataFrame and write DataFrame to Parquet files, parquet() function from DataFrameReader and DataFrameWriter are used to read from and write/create a Parquet file respectively. Parquet files maintain the schema along with the data hence it is used to process a structured…

Continue Reading PySpark Read and Write Parquet File

How to Load Parquet file into Snowflake table

Here, you will learn Parquet introduction, It's advantages and steps involved to load Parquet data file into Snowflake data warehouse table using PUT SQL and then load Parquet file from interstage to Snowflake database table using COPY INTO. Related: Unload Snowflake table to Parquet file Apache Parquet Introduction Apache Parquet is…

Continue Reading How to Load Parquet file into Snowflake table

SnowSQL – Unload Snowflake Table to CSV file

Snowflake data warehouse is a cloud database hence we often need to unload/download the Snowflake table to the local file system in a CSV file format, you can use data unloading SnowSQL COPY INTO statement to unload/download/export the data to file system on Windows, Linux or Mac OS. It doesn't…

Continue Reading SnowSQL – Unload Snowflake Table to CSV file

SnowSQL – Unload Snowflake table to Amazon S3

Using SnowSQL COPY INTO statement you can unload the Snowflake table in a Parquet, CSV file formats straight into Amazon S3 bucket external location without using any internal stage and use AWS utilities to download from the S3 bucket to your local file system. Pre-requisite Install Snowflake CLI to run…

Continue Reading SnowSQL – Unload Snowflake table to Amazon S3

SnowSQL – Unload Snowflake table to Parquet file

Using SnowSQL COPY INTO statement you can download/unload the Snowflake table to Parquet file. Unloading a Snowflake table to the Parquet file is a two-step process. First use "COPY INTO" statement, which copies the table into the Snowflake internal stage, external stage or external location.Use "GET" statement to download the…

Continue Reading SnowSQL – Unload Snowflake table to Parquet file

Download Snowflake table to local WINDOWS | Linux | MAC

As Snowflake data warehouse is a cloud database, you can use data unloading SQL COPY INTO statement to unload/download/export the data from Snowflake table to flat file on the local file system, Amazon S3, Microsoft Azure, Google GCP and more. In this article, I will explain how to export to…

Continue Reading Download Snowflake table to local WINDOWS | Linux | MAC

Spark Parquet file to CSV format

In this Spark article, you will learn how to convert Parquet file to CSV file format with Scala example, In order to convert first, we will read a Parquet file into DataFrame and write it in a CSV file. What is Apache Parquet Apache Parquet is a columnar file format that…

Continue Reading Spark Parquet file to CSV format

Spark Convert Parquet file to JSON

In this Spark article, you will learn how to convert Parquet file to JSON file format with Scala example, In order to convert first, we will read a Parquet file into DataFrame and write it in a JSON file. What is Apache Parquet Apache Parquet is a columnar file format that…

Continue Reading Spark Convert Parquet file to JSON

Spark Convert Parquet file to Avro

In this Spark article, you will learn how to convert Parquet file to Avro file format with Scala example, In order to convert first, we will read a Parquet file into DataFrame and write it in a Avro file. What is Apache Parquet Apache Parquet is a columnar file format that…

Continue Reading Spark Convert Parquet file to Avro

Spark Convert Avro file to Parquet

In this Spark article, you will learn how to convert Avro file to Parquet file format with Scala example, In order to convert first, we will read an Avro file into DataFrame and write it in a Parquet file. Read Avro into DataFrame What is Apache Avro Apache Avro is an…

Continue Reading Spark Convert Avro file to Parquet