I am running the Data Profiling Task on a table with 3 million rows and receiving an out of memory exception. I am pulling the data into a developer machine to do the profiling so the tables being profiled are on the local machine running SSIS and the machine has 8gb ram. Is there a way to regulate memory usage in the Data Profiling Task to mitigate this issue?

Hi, Can you share with me 1) how many columns do you have ? 2) Are you profiling the Candidate Key ? and how many keyColumns are you profiling ? Cheers

It is about 100 columns and we are profiling for candidate keys. the defualt profiling settings ran in 6.30 seconds but when adding the column pattern profiling checkbox, the timeout occurred.

There is nothing in the Data Profiling Task that would help you to consume less memory. It is just your situation where you have 8 GB of RAM (too low for what you need) only available, and you have 100 columns. There are two options for you to choose to proceed with: Add more memory, or do that on a more capable machine and/orDo the profiling in small increments, e.g. the 1st 20 columns first, then next 20 until all 100 are profiled. Arthur My Blog

how would I connect the columns back together to get a comprehensive view of the table? arent some of the profiling activities involved dependent on the column's relationships to each other? regarding option #1, can you point me to any benchmarking references for dataset size?

All the 100 columns cannot be candidates, narrow down to only those most possible thus making the data profiling leaner, then run. Adding memory is just like that: buying a bigger RAM chipArthur My Blog

thanks arthurz, for option #1 I meant any benchmarking in terms of data set size, not how to purchase more ram. but this looks good enough for the start, thanks.