Relying solely on expert judgement
It has been a common experience in my career to sit in a room with a team of engineers and project managers to estimate stories. The typical process is to read through a story discuss it and then ask each engineer how long they think it will take. The engineers guess some number of days, hours or story points. Then the number goes into JIRA somewhere.
The engineers are experts here, they have done this a lot of times and are much better at it than the average person on the street. But if all you are doing is asking for one number from an expert who isn’t trained in estimation, you will not get good estimates. The most effective estimates come from Counting and Computing with Expert Judgement as a last resort.
Using estimates in a group setting
Most backlog grooming and other estimation meetings are done in a group setting. Engineers make their estimates one at a time, then the team averages them together. This setup has two issues, first each engineer knows what the previous one estimated and will subconsciously adjust his estimate to match. Less experienced or less confident engineers will completely abandon their judgement and defer to more senior and louder spoken team members. Planning poker is a popular solution to this problem, but it still falls victim to the next problem below.
The second problem is time. You cannot make a good estimate in 5 minutes. Most estimation meetings run over time because the stories are incomplete and the team ends up discussing them. Each discussion has a risk of going off on a 10 minute tangent. Then with 10 minutes left on the clock the project manager wants to finish estimating 8 stories.
Not having historical data
I have worked with over 10 different teams in my career. None of them have had consistent historical data going back more than 3 months. A lot of agile estimation assumes teams have historical data they can use to calibrate their estimates. The reality is that teams do not have that data. Either the team does not track both the time to completion and the estimate for a story. Or commonly their is a new project manager every 6 months who implements a new agile process based on story points instead of time estimates.
To get good historical data you need to have consistent systems for estimation and data collection across teams. Teams are not a good level of granularity for historical data because they change all the time. Teams are dynamic managers change, engineers move to new jobs, ownership of a service moves to a new team. You won’t get consistent historical data if the process changes completely every time your project manager job hops.