It is better to increase the font size in small buttons. This makes the button label more readable. Although in general, it is questionable the use of small buttons. They are difficult to click by cursor or tap by finger on mobile.
Big buttons with a huge font size in them usually look out of proportion to the overall design and does not solve any problem. It is enough to make the font size slightly larger than the regular button.
If the button is to the right or left of the input, it must be the same height as the input. This simple rule should be considered at the design stage of the interface and create all the inputs and buttons of the same height.