This is mentioned in chapter 127 of the manga1, where Lala reminisced about the time she got lost when she was small, and her father (King Deviluke) found her. At that time, King Deviluke carried her back to the castle, since he was still at his normal size. Only later in the Galaxy Unification War that he used up too much of his power and ended up shrinking to the size of a small child.
1 This is during the journey to planet Mistletoe to find a cure for Celine. Lala was under effect of pollen and was drained of all her physical power. The reminiscence happens when she was carried by Rito on the back.
This trait of the Deviluke is again shown in chapter 44 of To LOVE-Ru Darkness where Lala herself turns into small child's body after exhausting all her power to protect Rito from Golden Darkness's rampage.