Spark Read Json From Amazon S3

Using Spark SQL spark.read.json("path") you can read a JSON file from Amazon S3 bucket, HDFS, Local file system, and many other file systems supported by Spark. Similarly using write.json("path") method of DataFrame you can save or write DataFrame in JSON format to Amazon S3 bucket. In this tutorial, you will…

Continue Reading Spark Read Json From Amazon S3

Spark Read Text File from AWS S3 bucket

In this Spark sparkContext.textFile() and sparkContext.wholeTextFiles() methods to use to read test file from Amazon AWS S3 into RDD and spark.read.text() and spark.read.textFile() methods to read from Amazon AWS S3 into DataFrame. Using these methods we can also read all files from a directory and files with a specific pattern…

Continue Reading Spark Read Text File from AWS S3 bucket

SnowSQL – Unload Snowflake table to Amazon S3

Using SnowSQL COPY INTO statement you can unload the Snowflake table in a Parquet, CSV file formats straight into Amazon S3 bucket external location without using any internal stage and use AWS utilities to download from the S3 bucket to your local file system. Pre-requisite Install Snowflake CLI to run…

Continue Reading SnowSQL – Unload Snowflake table to Amazon S3