Hi I draw alot of data from a DB2/AS400. Is it at all possible to draw only data that has changed or data that has been added since the last run. Thanks

There can be multiple ways of doing a Delta load in an ETL environment. To get you started, if your source data cotinues to change drastically since the last extraction, you can try bringing in all the deta into a Staging area, then do a JOIN with the existing data in your destination (in some SQL Server table(s)), and reload the data that has changed based on what attributes of source data you'd like to track for changes. However, if more than one-third of your data has changed, its suggested to do a FULL load (Truncate-and-Load), since the time taken by the process to complete in these cases will be comparable. However, if you are in a reporting environment where you maintain history of changes made to data overtime, rather than doing a hard-delete, you can do a soft delete by expiring the existing records and adding them again with changed attribute values, hence maintaining the versioning of records. Hope this helps. Cheers!! Muqadder.

I don't know - only you can answer that. Is there any information on the source data that identifies when it was last changed? If it doesn't, then there's no way to retrieve only the changed data from the source system, regardless of what database it's in. Muqadder's suggestions assume that you don't have this information. When you don't know what's changed, then your only recourse is to either truncate and load everything again, or do some kind of a comparison and delta yourself. In SSIS, this kind of thing is either done with reading in both tables and using a Merge Join (as Muqadder suggested) or by using the Lookup component to detect whether a row is already in the destination, and what it looks like. Talk to me now on

Thanks. I have tried both of the above. Just wanted to know if there was any other way. Long Shot!!