It matters as it is best and better to do it on a fixed time. Also, much of our daily activities if done on a routine fixed time will have much more power and importance. From eating to spiritual work, your body and mind learns and expects in time, it will create fruitful sessions, prepare beforehand and better everything. This is due to forming actual "habits" (the word lost weight due to artificial popularity), but really habits are simple brain connections and "wirings" that are physical. A habit is formed by repetition and very importantly, timing.
The above is to look for to perfect in time and not be worried about any output whilst you have other factors to consider developing first, such as quality and having some backend spiritual advancement.
A daily goal is your first priority here. Second, third, forth, etc. you can consider the timing on the hours.