First, good idea. I think you can take it a long way if you get the technology ended in place. I don't have an instant solution, and maybe someone will jump in with one, but let me help you a little bit with organizing thoughts on what you need to accomplish.
1) You need to host videos on the Internet within a secure directory so that everyone can't get to them
2) You need to receive payment so someone can gain access to the above videos
3) You need to provide a login and secured web pages for the above people to view the videos within the directory
The above would allow me to go to your website. Provide payment. And login to view the videos you want to provide me in return for payment.
You should look over the above and refine to your requirements.
From there, as you define the project details then you will want to start to further define how you will go about accomplishing each step required.
For starters, hosting the videos with youtube you mention the following above
Then when people pay for access we add their email address to the video, which allows them to see the videos. Private YouTube videos are only visible to people on the share list vis email address of Google+ profile.
I'm not familiar with how you are going to do that. But it sounds like a mess if you get 1000s of people signing up. Is this something you would have to do one by one, or is this something that can be automated?
First step. Hosting videos so restricted for paid access. This can be done by almost any hosting company (including probably the one you use). That is where I would start. Your current host and web provider are? Can they help?