Xjyuk

I am currently a fourth-year Electrical Engineering student. I have been a top-ranked student in our department. I have passed 3 courses with a professor in our department and have received a full mark in all three. I did participate a lot in all classes and had many discussions about the courses in his office hours. He knows me very well. I have also been the teaching assistant of this professor twice, for two distinct courses. I believe that I have also done a decent job as a TA for him. I also have worked in his research lab for a year now.

Recently I have asked this professor for a recommendation letter for graduate and he refuses, arguing he has decided to only give a recommendation letter to people who have written a research paper with him. By the way, he is willing to accept me as a master student, in his group but I prefer moving to another university.

My professor is new to the field I am currently working on and I believe it is hard for an undergraduate student to write a scientific paper in this situation.

I wanted to ask whether his refusal for recommendation letters is reasonable and also ask for some advice on how to handle this situation.

Edit:
May reasonable was not the best choice of word. I meant whether only giving recommendation letters to co-authors is something "common".

We can not say if they are reasonable. We only know the details you provide and.moreover, we are not judges.

But this is the wrong question. Why should it matter for you if we say the prof is reasonable or not? There aremany unreasonable people in the world. Among those are professors, students, astronauts, beggars, social workers, farmers,.. with one word: everyone.
So son't ask whether they are reasonable. The answer does not help you.

How to handle this: Ask another prof if avaiable.

Yes, the refusal seems reasonable in general. The professor is the one who should decide who he writes reference letters to, and if he has a policy of providing letters only to collaborators that is his right. Though unfriendly to undergraduate, it does have some merit: his letters can be trusted to a very high degree.

More importantly: any kind of hesitation from a potential reference letter writer should be an immediate warning sign that the person may not give you a good letter.

Conclusion: find another professor.

I recommend not limiting your options. Since as you said, you are the top ranked student in your department, finding another teacher who's willing to write you a good recommendation letter shouldn't be that hard. If course it only makes sense to seek someone who you know can write you a good letter that helps your cause.

I wanted to ask whether his refusal for recommendation letters is reasonable

My personal opinion is that if you worked in his lab for a year (satisfactorily), it is unreasonable to refuse a letter. All the more so if you had other good interactions (TAing, etc.) with him.

and also ask for some advice on how to handle this situation!

Unfortunately, my opinion counts for absolutely nothing. You cannot force this professor to write you a letter, and would be unwise to try. All you can do is find someone else. Preferably someone who can comment on your research, or at least someone who can confirm that your advisor almost never writes letters for undergrads (i.e., vouching that the lack of a letter from your advisor should not reflect poorly on you).