Ralf Rangnick's side dominated throughout but ultimately paid for their profligacy in a tense shoot-out when, after the first 15 penalties had all been converted, United youngster Anthony Elanga blazed over to send 9,000 Boro fans wild.
Jadon Sancho's 25th-minute goal, shortly after Cristiano Ronaldo had missed a penalty, was scant reward for United's superiority with Ronaldo and Marcus Rashford both culpable of failing to take golden chances.
Chris Wilder's Boro kept themselves in the tie, however, and equalised in the 64th minute when former United youth player Matt Crooks converted from close range.
Bruno Fernandes somehow struck the post with the goal gaping soon after that and United then battered away at Boro's defence in extra time all to no avail.
Boro then showed incredible poise with their penalties to send the 12-time FA Cup winners tumbling out.
It was a hammer blow for United who will wonder how they will not be in the hat for the fifth round after having 30 goal attempts to Boro's six.
"Incredibly disappointed, we should have won and killed off the game in the first half," Rangnick said.
"Crossbar, post, missed penalty and numerous chances. It should have been 2-0 or 3-0."
United almost took the lead in the opening minutes as Sancho clipped the top of the crossbar before Bruno Fernandes missed the target from close range. They then looked certain to go ahead when Boro's Anfernee Dijksteel clumsily brought down Paul Pogba to give Ronaldo the chance from the penalty spot.
But the Portuguese stepped up and dragged his right-foot effort wide of the post.
When Sancho controlled a Fernandes pass and his angled shot took a slight deflection to beat Boro keeper Joe Lumley it appeared that United were in control.
But Ronaldo, who was having an off night, wasted a great chance when he shot straight at Lumley while Rashford had a goal disallowed and was twice denied by the keeper.
The pattern continued in the second half with Rashford volleying over from Sancho's cut back and Ronaldo then clipping the side-netting with a snap shot.
Boro began to grow in belief and levelled in slightly controversial circumstances.
Duncan Watmore controlled a lofted pass at the far post clearly with the aid of his arm and then lofted goalwards for Crooks to finish. A VAR check then allowed the goal to stand -- courtesy of a tweak to the handball rule this season.
Boro were energised and had chances to go ahead before United took charge again with Fernandes guilty of the worst miss of the night as he was presented with an unguarded goal after a defensive mix-up, but his shot clipped the outside of the post.
Elanga had the chance to win it deep into extra time but headed straight at Lumley and his wild penalty summed up a frustrating night for United but a memorable one for Middlesbrough who kicked off the FA Cup weekend in style.
"The FA Cup is alive and kicking, especially in Middlesbrough," said Wilder, whose Sheffield United team won at Old Trafford in the Premier League last season.
"I told the players it was all about making memories."