Today I needed to add a new toolbar button to a custom Joomla! component. In the past, I’ve used a more convoluted method of overriding the submitbutton() (or Joomla.submitbutton) JavaScript function and triggering the modal popup manually, but stumbled upon a better solution. Instead of using the JToolBarHelper class to add a custom button, I used JToolBar: 
$bar =& JToolBar::getInstance('toolbar');
$icon_image = 'myicon.png';
$alt_text = 'My Label';
$page_url = JFilterOutput::ampReplace(
'index.php?option=com_mycomponent&view=myview&tmpl=component'
);
$width = 600;
$height = 400;
$bar->appendButton(
'Popup', $icon_image, $alt_text, $page_url, $width, $height
);
…of course switching in actual values for the variables. Since it opens an iframe popup menu, add tmpl=component in the $page_url variable so it omits the top navigation and whatnot. Also, note that I’m using JFilterOutput::ampReplace to convert & to & since appendButton doesn’t do that part for you. The info was gleaned mostly from here: http://docs.joomla.org/How_to_create_a_custom_button