Scheme is one of those languages that are easy to learn but hard to master. I'd say start with Scheme, then try Common Lisp, see if you like it. If you don't, you can always go back to Scheme. (Disclaimer: I'm a Schemer who does Clojure at work.)
One thing I always hated about CL is that it's not a very functional language, as in it gets in my way when I try to do things the FP way and immutably; not to mention Lisp-2 screws with my intuition of higher order functions. You should bear this in mind if you're a functional programmer at heart like I am. If you do end up going with CL, consider looking into fset, modf and Sycamore for persistent data structures, and folio2 for FP utility functions that augment Alexandria and Serapeum.