Evan,  This could be thermostat, wiring or relay issue. I would start by disconnecting the thermostat and valve wiring at the relay board and see if the problem goes away.   If it doesn't it is a relay issue.   If it goes away then it more likely is a wiring or thermostat issue.  You can jump the board out to troubleshoot.   If you have questions or need assistance please call Caleffi Tech Support 414-238-2360.