Instead of using a fileset for your code why don't you use the dirset, according to the ant-contrib documentation this should return directories.
So your for loop would look something like:
<ac:foreach target="trimmer" param="file" xmlns:ac="#unknown">
<path>
<dirset dir="="${directory}/SourceFolder/"" includes="*" />
</path>
</ac:foreach>