Are any parts of HTMLValidator Open Source?

Yes, the core validation system is entirely open source and distributed under the W3C license.
You can if you want entirely replace the validation system with your own, or use the validation code separately.

The source code is included inside the application bundle in various directories inside the Resources Directory.

The basic design of HTMLValidator is a slightly modified copy of the W3C validator with a user interface connected to it. The user interface aspects have been altered to make them fit better and the main scripts have been altered to make them work better in application format.