When I run your code, I get exactly what I expect:
declare @data varbinary(max)
set @data = 0xFFD8FFE0
select @data
declare @data1 varbinary(max)
set @data1 = 0xFFD8FFE00
select @data1
Gives
(No column name)
0xFFD8FFE0
(No column name)
0x0FFD8FFE00
Which is correct.
The problem is that you are expecting a "leading zero" on the first number, and it doesn't exist. When you store data in a VARBINARY column, it is stored as bytes - 8 bit unsigned quantities. but the data you enter is treated as pairs of hex digits, starting with the right hand end. So 0xFFD8FFE0 is stored as bytes:
FF D8 FF E0
and 0xFFD8FFE00 is stored as
0F FD 8F FE 00
If you don't provide an even number of digits, a "leading zero" is inserted in the data, as I would expect.