Half-time substitute Sebastian Tounekti reduced the deficit with a delightful curling effort, Benjamin Nygren poked home an equaliser in the 64th minute, and then Julian Araujo scored a dramatic winner in the seventh minute of stoppage time.
Celtic are three points behind William Hill Premiership leaders Hearts and one behind Rangers, with a game in hand over both teams.
In a period of weekend/midweek/weekend fixtures, next up is the visit on Thursday of German side Stuttgart for the first leg of the Europa League play-off at Parkhead. Hoops boss O’Neill told CelticTV: “Stuttgart. I haven’t had time to think about it.
“They were live on TV on Saturday. And my concentration was totally on Kilmarnock, so I couldn’t even be bothered watching it. I don’t mean bothered watching it. I just thought, ‘well, no, what’s the point in thinking about something that doesn’t exist for another few days?’
“So that was the case. I will get round to it now.”
O’Neill, in his second spell as interim boss this season after being Celtic manager between 2000 and 2005, admits every point is precious in a bid to keep the title at Parkhead while also battling on other fronts.
After taking on the Bundesliga side, Celtic host Hibernian, travel to Germany for the second leg of the play-off and then take on Aberdeen and Rangers away, before a Scottish Gas Scottish Cup quarter-final tie against the Light Blues at Ibrox. He said: “Still a long way to go, but we’ve got the three points in the board, and naturally, it helps.
“But, again, no room for manoeuvring. No wriggle room, all that type of stuff. It’s as if the players highly believe that there isn’t any wriggle room, and we won, great.
“And this is the start of a massive couple of weeks for us now.”