Click here to Skip to main content
15,672,318 members
Please Sign up or sign in to vote.
1.00/5 (2 votes)
See more:
how should i create ETL test cases...
what we do is get all data from test server, process and load into some other table.
i need some sample code to test for such scenarios

What I have tried:

1. how to create unit test project for etl job
2. how to test if data is being loaded or not
Updated 5-Sep-21 22:06pm
James Walsh Jr 3-Sep-21 20:15pm    
Without knowing what you're using for your ETL process, it's impossible to answer this question.
What ETL tool are you using, what platform? What have you tried? Are you getting a specific error?
Alot more detail is required before anyone can help you.
PIEBALDconsult 4-Sep-21 0:05am    
No idea, but you kinda don't.
I've used SSIS quite a bit, but otherwise I prefer to write my own ETL tools.
The situation I'm in now is that my boss has asked me to verify that other team members have "loaded the data correctly". I could just weep.

1 solution

You create "expected output"; e.g. a table, work book, flat file, csv, whatver. Then you run your ETL and compare it to "expected". They should match for a successfull test; you do a tablec compare; file compare; etc. using any "compare tool", including SQL. That's how audit groups do it. The "expected output" is retained for any future regression / unit testing.
Share this answer
PIEBALDconsult 6-Sep-21 12:18pm    
But that might not flex the same bits of functionality which is to be tested.
And there could be a problem in the code you use during your "test" which is not present in the code you are meant to be testing -- leading to a false-positive or -negative result.
Gerry Schmitz 10-Sep-21 12:42pm    
It's an "extract", then an (optional) translate, then a load. It's not that complicated. The "audit groups" I worked with would create master files of how they "should look" after a series of transactions agains a "known" starting master. Finance. No idea where you're coming from.
PIEBALDconsult 10-Sep-21 12:47pm    
Loading from "master files" as you suggest is not the same as loading from a database as the OP implies -- you would be testing a different code path.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900