I don't think this has anything to do with primary key, unless you are saying DepartmentId is a primary key.
My suggestion would be to add to your DropDownListFor data a default option of "Select" and set the value of that Select to 0. Since your field is an integer, and assuming its required as such in the database than its going to be 0 anyway. If 0 is a possible value, then set the value to say...-1.
So your code would look like
var selectData = new List<SelectListItem>();
selectData.Add(new SelectListItem { Text ="Select", Value = "-1"});
selectData.Add(new SelectListItem { Text ="Actual Selection", Value = "Actual Value"});
@Html.DropDownListFor(m=>m.DepartmentID, selectData, new{@class=""})
This way, if no selection is made, you know that -1 means they didnt choose any values.
The other option is to pass it in as -1 and when the validation runs, use the null coalescing operator to default the value to 0 if what comes through from the model is null.
Ex:
var safeDeptId = (DepartmentID ?? 0);
That way if no option was selected, your defaulting to 0 and shouldn't get the error indicating you are missing a cast.
Just some ideas on how to solve your problem.