Isner had 21 aces in his Davis Cup debut, but Troicki converted his fourth match point with a crosscourt volley after 3 hours, 36 minutes on the clay court at Belgrade Arena.
Sam Querrey faces second-ranked Novak Djokovic in the second singles.
Story continues below ↓advertisement | your ad hereThe match marks the first time since 2000 that neither Andy Roddick nor James Blake has played for the U.S. team.
Serbia is trying to move beyond the first round in the World Group for the first time.
© 2010 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.La Liga round-upDjokovic to lead Serbia against U.S. in Davis Cup