Rule of thumb is, who creates memory has to delete it. So when you remove an vector you must delete the memory.
But you have the additional problem, that your vector may be copied or assigned somewhere deep in the vector implementation and so you may have 2 vectors with the same memory. Maybe it helps to overwrite these operators in which you copy the memory of the sourse vector.
I think you dont need to allocate the Item with new, but can use a typed vector