Comments by Silpa Silpa (Top 2 by date)

Silpa Silpa 27-Aug-22 5:05am View

Oh ok thank you for your confirmation.. i am not expecting some one to write full code for me.. i am using sql and i also know how to filter in sql but in this code they are asking about spark api i don't know how to filter using in spark api scala. That thing i don't know..so i posted full question for clarification only..i am just searching for example ..not for full code.. thankyou.. i think i reached wrong site...

Silpa Silpa 27-Aug-22 4:33am View

: Spark Scala coding exercise based on below requirement.
Expectation is to implement all the requirements using latest SPARK API (not sql) and functionalities as per our cluster upgraded version (spark version - 2.4.0, scala -2.11.8).

Input:
1. Given below input Partner data in comma separated file. (file name - Partner.csv).
2. Given List of Invalid Party Ids.

Requirement:
1 Read the input Partner csv file and Invalid party ids list as Dataframes using spark APIs.
3. Retrieve the latest record per party id based on the Last updated Date from input Partner Data.
4. Retrieve the valid Party ids from the Latest record DataFrame based on lookup from Invalid Party DataFrame.