have you considered jazz or stuff like "jamming" ala bands like phish or goose or similar, just as something to consider? That's what comes to mind for me when I think improvisation in live music.
It's really a good question that I wish I had some answers for, but I am not enough of a musician to tell you. I'm hardly a musician at all. You could pick apart note by note what they're doing and try to see the relations a bar at a time, but this seems tedious when there might very welll be some more reasonable and compact explanation out there for what you're trying to understand.
when you find out what you're looking for, would you post your findings?