United have conceded 62 goals in 31 Premiership games and nearly threw away a 2-0 lead at home to Leicester City on Monday before winning 3-2.
Caldwell, on loan from Newcastle, said: "There were a few things wrong with our defending and we have to work on that. We have the quality to get out of this position but we showed our frailties against Leicester.
"We were under pressure late on and defended deeper. We know we have to tighten up because if we continue to play like that we will concede goals."