dynamically set onclick event not working :(

Hello! I have a small script which dynamically creates select tags (they are placed within divs). Each select has a + and - sign next to it, and if you click the + it adds another select below that one, or if you click the -, it removes that current select.

I've made it so that if there is only one select, it can't be removed. So i made it like this, when i create a select, the remove button is inactive. When i click to add another select button, it becomes active.

The problem is that the onclick event is not working with any browser except Opera. This is the code:

function createDiv(divNum){
  // code that creates new selects here
  if (divNum > 1){ // if this is an additional select
    var prevNum = divNum - 1;
    var prevRemove = "remove" + prevNum; // name of the button
    document.getElementById(prevRemove).src = "minus.png";  // change icon so that it shows it is enabled - this works
    document.getElementById(prevRemove).onclick = "removeDiv(" + prevNum + ");";
  }
}

function removeDiv(divNum){
  var container = document.getElementById("container");
  var removedID = document.getElementById("div" + divNum);
  container.removeChild(removedID);
}

It works in Opera but no other browser. Any help is appreciated, thank you!