If I understand the question correctly you could use an
unique index
to enforce the logic.
Consider the following test
CREATE TABLE UniqueTest (
col1 int,
col2 int
);
CREATE UNIQUE INDEX U_UniqueTest ON UniqueTest (col1, col2);
INSERT INTO UniqueTest VALUES (1, 1);
INSERT INTO UniqueTest VALUES (1, 2);
INSERT INTO UniqueTest VALUES (1, NULL);
INSERT INTO UniqueTest VALUES (NULL, 1);
INSERT INTO UniqueTest VALUES (NULL, NULL);
INSERT INTO UniqueTest VALUES (1, 1);
INSERT INTO UniqueTest VALUES (1, 2);
INSERT INTO UniqueTest VALUES (1, NULL);
INSERT INTO UniqueTest VALUES (NULL, 1);
INSERT INTO UniqueTest VALUES (NULL, NULL);