Top-seeded Ivanovic advanced 6-7 (4), 7-5, 6-1 over the 82nd-ranked Vinci and next faces former No. 1 Amelie Mauresmo, who endured a marathon three-set win over fellow Frenchwoman Julie Coin.
Ivanovic was down a set and gave Vinci double matchpoint at 15-40 in the 10th game of the second. The 21-year-old Serb put everything into her serve and ground strokes and forced enough errors from Vinci to stay in the game.
She pulled back to 5-5, broke Vinci and then served out to love, leveling at a set apiece.
She then won nine of the first 10 points of the 27-minute deciding set and raced to a 5-0 lead.
Ivanovic said surviving the close contest gave her confidence ahead of the Australian Open starting Jan. 19 in Melbourne, where she reached the final last year.
"It was really tough. She was playing really well, mixing it up a lot. I wasn't as aggressive as I should have been,'' she said of Vinci. "I tried to be fitter and tougher mentally than she was and it paid off. So that gives me confidence.''
Ivanovic, who spent 12 weeks at No. 1 last season, also struggled at times in her first-round win over Petra Kvitova - her first match of the season - and was pushed to the wire by Vinci, a qualifier for the main draw.
She hadn't dropped a set in two previous wins over the Italian. But that streak was soon ended. After fending off four set points in the ninth game of the opening set, then recovering a break, Ivanovic was outplayed in the tiebreaker.
Ivanovic was only a point from being the second major casualty of the inaugural Brisbane tournament. Her compatriot, and men's top seed, Novak Djokovic was bundled out in the first round.
Instead, Ivanovic will meet 2006 Australian Open and Wimbledon champion Mauresmo on Pat Rafter Arena on Thursday.
Mauresmo recovered from 2-5 down in the third set to beat Coin 5-7, 6-2, 7-6 (11) in her longest ever tour match.
The 29-year-old former Mauresmo saved three match points and need five of her own in the 3-hour, 14-minute second-round win over 95th-ranked Coin, who only got a place in draw when Maria Kirilenko pulled out.
"I felt it was long, so long,'' said Mauresmo, whose previous longest match in 13 years on tour was 3:07 in her win over Mary Pierce at the WTA Championships final in 2005. "It's tough. I'm tight everywhere. A lot of work for my physio.''
Mauresmo double-faulted on her first match point at 6-4 in the tiebreaker.
She had served for the first set at 5-4 but was broken twice.
In the end, fifth-seeded Mauresmo converted five of her 20 breakpoint chances and fended off 11 of 15 she faced.
"Too many mistakes, too many missed chances - yeah, it was a strange one,'' said Mauresmo, who had to ask how many match points she had but knew about her 10 double-faults.
In an earlier second-round match, Italy's Sara Errani beat Bulgarian Tsvetana Pironkova 6-7 (3), 7-5, 6-3.
On the men's side, No. 2 Jo-Wilfried Tsonga - who lost to Djokovic in the last Australian Open final - No. 3 Fernando Verdasco and No. 8 Radek Stepanek all advanced.
No. 5 Tomas Berdych 7-6 (9-7) 6-3 by 19-year-old Japanese player Kei Nishikori.
Ernests Gulbis lost 6-3, 6-4 to Paul-Henri Mathieu a day after upsetting top-seeded Djokovic.
Mathieu was one of three Frenchman going through, including No. 6-ranked Tsonga, who was broken three times in the first set but recovered to beat Jarkko Nieminen of Finland 0-6, 7-6 (1), 7-6 (5).
He'll play another Frenchman, seventh-seeded Richard Gasquet, in the quarterfinals.
Florent Serra beat Austria's Jurgen Melzer 6-4, 6-3.
Fernando Verdasco of Spain beat Croatia's Mario Ancic 6-2, 6-3 and Stepanek of Czech Republic defeated Michael Llodra of France 7-6 (2), 6-3.
Ivanovic opens season with win
Eguren Calls On Villarreal To Act, Not React
Mauresmo rallies to beat Coin in Brisbane
