Xjyuk

Followers of Vivekananda and reformist Hindus today such as Arya Samaj believe that caste is determined by someone's external behavior. For example, if someone is pious, he should be a Brahmin.

Has any ancient or medieval Vedic scholar refuted this guna-based theory?

Has any ancient or medieval Vedic scholar refuted the guna-based varna theory?

Yes, medieval Sri Vaishnava scholar Vedanta Desikan has refuted the guna-based theory in his work called the Rahasyatraya Sara.

In the section titled "Does prapatti (surrendering to Lord Vishnu) entitle a shudra to be considered a brahmin?" he says:

It is said: "Those shudras who have devotion to Bhagavan are not shudras, they are who are devotees of Bhagavan are brahmins." ... From this, men of poor understanding might ask, 'Are not the devotees of Bhagavan of one and the same caste?' If this view were accepted, it would be in conflict with all the shastras that prescribe the respective course of right conduct for the castes. - page 300

Shastras such as the dharmashastras which have verses like:

Manusmriti 10.5: Among all castes, those only who are born of consorts
wedded in the natural order, as virgins of equal status, are to be
regarded as the same (as their father).

Āpastamba (2.13.1).—‘Sons begotten by a man who approaches in the
proper season a woman of equal caste, who has not belonged to another
man, and who has been married legally, have a right to follow the
occupations of their castes.’

Viṣṇu (16.1).—‘On women equal in caste to their husbands, sons are
begotten who are equal in caste to their fathers.’

Yājñavalkya (1.90).—‘From women of the same caste as their husbands
are born sons of the same caste.’

Baudhāyana (1.17.2).—‘Sons of equal caste spring from women of equal
caste.’

So how do we reconcile these verses and other verses that appear to state that a shudra is a brahmin or a brahmin is a shudra by conduct alone?

First, Vedanta Desikan says that the caste doesn't change, but that the shudra who has brahminical qualities should be respected like a brahmin, and the brahmin who doesn't have devotion to Bhagavan is as good as a shudra.

In such passages as this: - "He should be considered as a Shudra", the great sages called men of higher castes by the words applicable to lower castes and vice versa merely to indicate the degree of consideration or respect to be shown. - page 301

Second, Vedanta Desikan shows that there is a difference between guna of the mind (manas) and guna of the body (sthula sharira):

Owing to the preponderance of such qualities as sattvam, in the body, a man is entitled to be called a Brahmin, a kshatriya, and the like. But this is different from the praise of being a Brahmin that is often given in certain passages, owing to the quality of sattvam and the like in the mind. The qualities of caste pertaining to the body which are due to the special qualities of the body arise even at the time of birth and remain until death. The Brahmin-like qualities which arise out of the preponderance of sattvam in the mind may be present in all castes. In Prahlada [an Asura], they are present even at the time of birth. In others, owing to such specific causes as contact with acharyas, these mental qualities are acquired later.

If in a Brahmin are found such features as are incongruous with his
caste, it will meet with censure that, born in a caste which is
declared by the shastras as being capable of such virtues as self
restraint and mental serenity, he lapsed from the condition suitable
to him. - page 306

So, caste is determined by guna of the body, and not the mind.

If caste was determined by external behavior solely, then it would be almost impossible to determine someone's caste as illustrated by this question.

And finally, Vedanta Desikan also says that certain scriptures that claim that caste can easily be changed in the same life are heretical and non-Vedic:

Those who abide by the shastras should not believe in the elevation
from lower castes stated in the deceptive shastras of heretics. -
page 303

This shows that the belief in a guna-based varna theory was considered heretical and non-Vedic as old as medieval times.

Take a look at my answer here that shows that the Vedas declare a caste system based on birth.