Sounds like you could be missing a few indexes on your table.
Try running your select queries in a query editor of MSSQL Management Studio with the show execution plan option set. it will tell you where the time is spent and suggest missing indexes.
http://technet.microsoft.com/en-us/library/ms178071(v=sql.105).aspx[
^]
Essentially you can right click on the line with missing index is causing ... text and select functionality something like 'copy definition to new query editor' there are only a few options.