Truth is I would fall in line with Tartar here, I doubt the cops even give a damn about what they consider "petty" things like this. Not saying it's the smartest thing to do but I know how I am, these things piss me off and there would be some sort of personal confrontation the next time I saw him. Make sure your wife and others in the house witnessed you not only asking him to not violate your private property before you go talk to him but also witnessed him continuing to do so. If they didn't make sure they are well coached before you go meet up with him. That way if it gets ugly it will be your word against his, even if you hit him first make sure you deny it and were only defending yourself, just say you went over to ask him why he continues to drive through your property after you advised him to stop. Even when you tried to settle it like two adults he got angry and got physical with you, you had no choice but to defend yourself.
Of course the smarter thing to do would probably be go talk to the lot owner that he is doing work for, chances are if the guy is that lazy to drive properly the work he's doing will also be sloppy and halfassed. Explain to your neighbor what has been going on and see if he will talk to the guy for you and ask him to knock it off or hire someone else. Unfortunately though dealing with assholes like this you usually need to make a strong stand, they don't deal logically like most, see paragraph one for what I think is your best bet. Good luck.