The City Pier is closest to town; you can actually walk to Carlos & Charlies from there. The International Pier has the advantage of having the duty free shopping mall at the end of the pier; and, of course you are much closer to Chankanaab Park. In any event, a taxi ride to downtown is only $5 for the entire cab, not per person.