Here is something peculiar. Irregex-match uses irregex-match/chunked which has two branches, one with dfa-match/longest and another with irregex-nfa. But the irregex-search/matches used by irregex-search has three branches: dfa-match/longest, dfa-match/shortest and irregex-search/backtrack. However, it is not a difference between these branches that breaks irregex-search. With some bronze age debug prints we have:
$ guile -l irregex.scm
[...]
scheme@(guile-user)> (define (imsis re str) (irregex-match-substring (irregex-search re str)))
scheme@(guile-user)> (imsis "(a|ab){0,3}" "abab")
is/m->backtrack $1 = "a"
scheme@(guile-user)> (imsis "(ab|a){0,3}" "abab")
is/m->backtrack $2 = "abab"
scheme@(guile-user)> (imsis "(a|ab)*" "abab")
is/m->dfa is/m->shortest $3 = "abab"
scheme@(guile-user)> So both range iteration versions take the backtrack branch, but only the prefix-first version breaks irregex-search. It would seem that the problem might be somewhere within irregex-search/backtrack.