I agree with Logan - First, I'd like to give you a little background - I used to be a senior manager in a company where I was a shareholder. I am not longer active in that company, but I am still an investor. The business was tricky, but we did love what we were doing - we started with 6 employees and when I decided to explore other fields / businesses, we had close to 60 employees.
Going back to your scenario and why I said I agree with Logan - your company/ team is still growing and it is an opportunity for you to keep it growing - before you do so, you will need to rally your good people around you, because when you grow your team, you will need to delegate some tasks. So, you can start off by picking your leaders - depending on the complexity of the tasks, you may need 1 leader to 4-5 employees. If the tasks are not too complex, you may have 9 - 12 or 16 - 18 employees per leader. From there, you can solicit the help of your leaders to come up with a performance metrics for their team - then, you will come up with performance metrics for your leaders. Once you have that in place, you can discuss the metrics with your leaders and they in turn can discuss it with their team members. You may also opt to sit in those discussions. During those discussions, you'll have an opportunity to ask feedback from all employees. Afterwards, you can use the same performance metrics to evaluate performance and determine salary increases, promotions, etc.
Benefits to having the performance metrics:
1. You and your team will have a clear GOAL
2. Depending on how they want to grow in the company, your employees will know what is expected of them and what they will achieve if they deliver
3. You will have a qualitative and quantitative means of evaluating and compensating your employees
Regarding your plan to talk to the 4 "problematic" employees or their gang leader, if you only talk to them and give them feedback, they might feel singled out - employees talk - oh how they talk - and when they tell others, other employees may feel left out or resentful that their take on things were not considered as well. These are just some of my thoughts
I know I may have gone overboard, but I really know what you are going through, which I why I thought to share. Goodluck!