Generally speaking, there is no legal issue of taking a car hire into Northern Ireland from the Republic.  Most car hire companies require you to let them know that you intend to and SOME companies charge a fee (typically, about 30 Euro) -- in order to cover the cost of Road Service, should you have a break down.  You'll have to check with your car hire company to see if they charge that fee.

There are no longer any border check points and in many areas, the only indication that you will have that you have crossed the border will be changes in road signage -- In the North, roads are typically A and B ( such as A24, or B122).  In the South (Republic) they are N and R ( N72, or R 583).  In the North, posted speeds are in Miles per hour and distances are given in miles, whereas, the Republic uses Kilometers per hour for posted speed limits and kilometers for distances.

VERY often, the roads in the North are of better surface quality than those in the Republic, too.    Roads in the Republic are Victims of Budget Priorities, I guess. 

The BIGGEST problem of driving a Hire Car from the Republic in the North is that the North utilizes MANY Traffic Speed Camera (Remote Radar) and cars from the Republic display ONLY in KPH and Kilometers --  

You will want to know the key 'Conversion Numbers':