How to update when insert completed...

Re-indenting your code so it is clearer:

create Proc insert_Returns
@RID int output,
@ReturnsType int,
@CSPrefix varchar(50)
AS
begin
   if not exists(select ReturnsType from tbl_ReturnSetings where RID=@RID)
   begin 
      INSERT [dbo].[tbl_ReturnSetings](
	[ReturnsType],
	[CSPrefix])
      VALUES(
	@ReturnsType,
	@CSPrefix)
      SELECT @RID=SCOPE_IDENTITY();
   end
   else
   begin
      UPDATE tbl_ReturnSettings
      SET CSPrefix = @CSPrefix 
      WHERE RID=@RID
   end
end

Makes it a bit more obvious.

But what you are asking for is pointless - your SP does an INSERT if no record exists, or does an UPDATE if it is already present.

But your insert looks to use an IDENTITY field - in which case you cannot provide an ID value (So are presumably passing a negative value to ensure it does an insert). And if it does an insert, then there is no point in doing the update, because it updates the field you just inserted with the same value are are updating...

I think you need to sit down and think about what you are trying to do, because I'm pretty sure what you have asked for is not it...

why are you setting @RID output when you are using it in your where statement.
don't you think you should pass it to you SP.

Try below code..

create Proc insert_Returns
@RID int,
@ReturnsType int,
@CSPrefix varchar(50)
AS
begin
   if not exists(select ReturnsType from tbl_ReturnSetings where RID=@RID)
   begin 
      INSERT [dbo].[tbl_ReturnSetings](
	[ReturnsType],
	[CSPrefix])
      VALUES(
	@ReturnsType,
	@CSPrefix)
      SELECT @RID=SCOPE_IDENTITY();
   end
   else
   begin
      UPDATE tbl_ReturnSettings
      SET CSPrefix = @CSPrefix 
      WHERE RID=@RID
   end
end

I think now it should work fine for you...