Hello, I have been looking for an answer to this and was not able to find one since all of them stated that you needed extra IPs. I've done this before and just forgot how to do it, but finally remember. Without further adeu..
For this we will assume that your domain is www.mydomain.com.
Also keep in mind that while you are a reseller, you can only go so far with completing these tasks, some of them requires our services, which we will happily provide.
If you are a dedicated server customer, then you can do all of the necessary steps listed below to properly set up private nameservers for your customers and not use any extra IPs.
Let's get started.
1. Login to your registrar panel (Godaddy, Enom, etc..)
2. Select the domain you want to use and find where you can 'Register New DNS'.
3. Put in ns1.yourdomain.com and ns2.yourdomain.com in the respective fields.
4. Put in the IP addresses (either yours or the ones provided to you by your host).
5. Hit the save/submit button.
NOTE: If you are a regular reseller, the rest is done by your host on their end, so notify them that you have done your part. If you run a dedicated server, read on.
6. Login to your WHM panel as 'root'.
7. Go to 'Reseller Center', select 'mydomain.com', click on the 'Edit Domains/Privilages' icon.
8. When the page loads, scroll down and put in the new nameservers, in this case ns1.mydomain.com and ns2.mydomain.com.
9 Save them and scroll down to 'DNS Function' and select 'Edit DNS Zone'.
10. You will notice that the new nameservers are now located in the top 2 text fields. Scroll all the way down to where it says 'Add new A entries'.
11. In the first text field put in the nameserver and in the second empty text field put in the IP address. So, it will look something like this:
ns1.mydomain.com 14400 IN A 1.1.1.1
ns2.mydomain.com 14400 IN A 1.1.1.2
12. Click Save and you are done!
Please remember that it may take some time for DNS to propogate.
Also, make sure to use IP addresses that you received from your host/data center.
If I missed something please feel free to comment. I'm pretty sure this is how you do it.