I've been obsessed with these jackets for forever! I love the "cool vibe" they give off. They're casual, but definitely not boring. I've seen people wear them in so many ways; they've even been worn uncasually.
I would wear a varsity jacket with ripped jeans, a t-shirt, and combat boots,
Varsity Jacket Love by dianeo
or a graphic printed t-shirt tucked into a bodycon miniskirt, along with sneakers or combat boots (as you can see, I'm obsessed with combat boots too!).
Here are some other ways people have worn varsity/letterman jackets: