As per the current setting enabled all the look up fields will show the recent items, but it should allow Recent Items option - conditional setting, Few of the fields need to show recent items and other fields don't required to show the recent items.
What is the underlying problem do you intend to solve with this idea?
some of the look up fields not required to show recent items
How is the problem being addressed today, if at all?
Condition based setting not available due to that not using this feature