Tuesday's 2-1 defeat at Leicester summed up the Whites' inconsistent beginning to the season and left them 10th in the table - a position not deemed suitable by some.
Grayson told the Yorkshire Post: "Where do people want us to be? Do they want us to be 10 points clear in the Championship, while being one of three teams promoted from League One last season?
"I've got an honest bunch of players. A lot of them were playing in League One last year and some have come in on free transfers. We haven't gone out and spent millions of pounds.
"I don't see how people expect us to be walking clear at the top of the division. We're in a great position. People can talk about us winning championships, promotion and European Cups but that's all in the past.
"This club now are in a position where me, my chairman and my staff think we're in the right place - but knowing that we can improve."
Source: PA