Mat cutting is difficult to do well in general, which is why you probably haven't been satisfied with (I'm guessing) what you've gotten from big box framing stores, who hire teenagers who don't give a shit. It either takes specialized automated equipment operated by said flunkies, or specialized hand-cutting tools requiring a lot of practice and skill. For customized cuts like what you're describing, you're probably going to have to pay a lot, because there's not a great automated way to do it; that will most likely have to be manually cut.
My advice would be to go to a small mom-and-pop framing shop with an old guy who appears to take pride in his craft, and prepare to pay for quality.