Spark – How to get current date & timestamp

Spark SQL provides current_date() and current_timestamp() functions which returns the current system date without timestamp and current system data with timestamp respectively, Let’s see how to get these with Scala and Pyspark examples.

current_date() – function return current system date without time in Spark DateType format “yyyy-MM-dd”

current_timestamp() – function returns current system date & timestamp in Spark TimestampType format “yyyy-MM-dd HH:mm:ss”

First, let’s get the current date and time in TimestampType format and then will convert these dates into a different format. Note that I’ve used wihtColumn() to add new columns to the DataFrame


import spark.sqlContext.implicits._

// Get current Date & Time
val df = Seq((1)).toDF("seq")

val curDate = df.withColumn("current_date",current_date().as("current_date"))
    .withColumn("current_timestamp",current_timestamp().as("current_timestamp"))
curDate.show(false)

Yields below output


// Output:
+---+------------+-----------------------+
|seq|current_date|current_timestamp      |
+---+------------+-----------------------+
|1  |2019-11-16  |2019-11-16 21:00:55.349|
+---+------------+-----------------------+

Now let’s split the date & time into a separate column from ‘current_timestamp’ column and format the date into ‘MM-dd-yyyy’


curDate.select(date_format(col("current_timestamp"),"MM-dd-yyyy").as("date"),
    date_format(col("current_timestamp"),"HH:mm:ss.SSS").as("time"),
    date_format(col("current_date"), "MM-dd-yyyy").as("current_date_formateed"))
    .show(false)

Yields below output


// Output:

+----------+------------+----------------------+
|date      |time        |current_date_formateed|
+----------+------------+----------------------+
|11-16-2019|21:00:55.705|11-16-2019            |
+----------+------------+----------------------+

Complete Example for reference


package com.sparkbyexamples.spark.dataframe.functions.datetime

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object CurrentDateAndTime extends App {

  val spark:SparkSession = SparkSession.builder()
    .master("local")
    .appName("SparkByExamples.com")
    .getOrCreate()
  spark.sparkContext.setLogLevel("ERROR")

  import spark.sqlContext.implicits._

  // Get current Date & Time
  val df = Seq((1)).toDF("seq")

  val curDate = df.withColumn("current_date",current_date().as("current_date"))
    .withColumn("current_timestamp",current_timestamp().as("current_timestamp"))
  curDate.show(false)


  curDate.select(date_format(col("current_timestamp"),"MM-dd-yyyy").as("date"),
    date_format(col("current_timestamp"),"HH:mm:ss.SSS").as("time"),
    date_format(col("current_date"), "MM-dd-yyyy").as("current_date_formateed"))
    .show(false)
}

The complete code can be downloaded from GitHub project

Happy Learning !!

Naveen (NNK)

Naveen (NNK) is a Data Engineer with 20+ years of experience in transforming data into actionable insights. Over the years, He has honed his expertise in designing, implementing, and maintaining data pipelines with frameworks like Apache Spark, PySpark, Pandas, R, Hive and Machine Learning. Naveen journey in the field of data engineering has been a continuous learning, innovation, and a strong commitment to data integrity. In this blog, he shares his experiences with the data as he come across. Follow Naveen @ LinkedIn and Medium

Leave a Reply

This Post Has 2 Comments

  1. Anonymous

    page scroll automatically

    1. NNK

      Apologies for the inconvenience. I’ve changed the layout now and hope the issue fixed. If not, could you please let me know what browser you are using?