You'll want to use a join instead. Essentially, if I understand your attempt at SQL correctly, you want to use the DebitAccNo from a matching record if it exists and if it does not you want to use the passed in parameter. Use LEFT JOIN for this.
SELECT id, COALESCE(t2.DebitAccNo, @DebitAccount)
FROM OUT_ITC_Final t1
LEFT JOIN AP.OutwardInstFinalID t2 ON t1.OutwardInstFinalID = t2.OutwardInstFinalID
WHERE t1.AutoReturnStatus = 7004
Something along these lines.