Well this was fast, but I found out the problem, the unit must be able to repair as well as construct. I didn't make the building repairable, only constructable, but for anyone who was interested in making it so only 1 worker could build a building, you could use this if you wanted to make it work, but if a worker gets distracted from the project, he can't resume it.