Yes, it took a lot of reading of various articles but I managed to figure it out.
I'll try to recap here - hopefully I don't forget anything.
1. On the workflow tasks, create a field. (I made it hidden on the Task form) Assign the ID number of the Document set to that field as the task is created.
2. Create the view in the Tasks list you want to use. (filter it so that it only has the workflow you want)
3. On the Document Set Welcome Page add the Tasks list (set the view to the one you want)
4. On the Document Set Welcome Page - in Sharepoint Designer. Select the web-part of the tasks list.
4a. Go to Parameters in the ribbon menu. Add a Parameter make the source 'Query String'. Make the Query String Variable 'ID' (without quotes).
4b. Go to Filter in the ribbon menu. Select your field you created on the task and make it equal to the Parm you created. *You can optionally filter by the workflow name as well if you didn't do so on the Task view.