You don't have to "remove" anything, unless you want to lie to yourself. Instead, you should face it: if you define a floating point value by zero, the result is -Inf or +Inf. In integer division by zero, this is invalid operation. You can even work with those Inf values. For example, you can compare + or -Inf with any other non-NaN value using <, <=, !=, > or >= operators and get a valid result of comparison, you can multiply it by any number, and a lot more. These are perfectly valid values, as soon as you understand their meaning.
These values, as all the presentations of floating-point type are represented in CPUs instruction sets
and standardized under IEEE 754. Please see:
As you can see, the "normal" floating-point values are complemented with +/-Infinity and NaN (Not a Number) values. If you think well about it, you will find many great uses of all these values.