Click here to Skip to main content
15,891,787 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I have a scenario Today where I am stucked Up little bit. I have data like this below in my Source Oracle Source Table(Sample Data)
ID,NAME,SALARY,BIRTHDAY
1,ABHIJIT,2000,17/12/1990
2,ROHIT,-2000,13/11/1988
3,MOHIT,500,2075-575-43

Now Salary in the 2nd Row is negetive and BIRTHDAY in the 3rd Row is an Invalid Format(Valid is MM/dd/yyyy). Both 2nd and 3rd Row should go to INVALID_EMPLOYEE and 1st record should go to VALID_EMPLOYEE.

In the Source File Date Format is coming as dd/mm/yyyy. Which I have to convert to MM/dd/yyyy format and also have to check whether Date Format incoming in Source File is dd/mm/yyyy or not.

Salary should not be less than 0. Source Table All columns are in String and in Target Table ID is Integer,NAME as VARCHAR2(255), SALARY is NUMBER and BIRTHDAY is DATE.

I have handled all this in my Project ETL Tool. So I am trying to push all this in Query to improve performance.Any help is much appreciated

What I have tried:

I have done already those using my ETL.
Posted
Updated 17-May-18 3:13am
v2

1 solution

Don't try to push this to the DB - it almost certainly won't improve performance, but actually make it worse, because you need to set up a round trip to the DB to validate it, and that's going to be slower than doing validations locally when you insert values.

In particular, your date is a problem, because it shouldn't be possible to get an invalid date as far as the DB - all your queries should be using parameterised queries which transfer a validated DateTime object - the only way to get a "bad date" to the DB is to pass it as a string, and that implies string concatenation which leaves your whole system vulnerable to SQL Injection problems which can damage your whole DB.

And then there is the separation of concerns POV: a negative salary is a business layer concern, not a DB concern, and again, should be validated and dealt with at the business or even presentation layer, not fed to the data layer, by which time it's too late to do much about it.

I think you need to rethink this - and probably a fair amount of the rest of your application if SQL Injection is possible.
 
Share this answer
 
Comments
MadMyche 17-May-18 10:12am    
Exactly. The database's job is data CRUD; it is the application using the DB's job to validate the data. Adding the validation to the DB layer is just shifting the responsibility, and IMHO the wrong way. I would be shifting it the data source to give you the correct information.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900