Disclaimer: I've never looked into the internals of the Vita, if not just because A) I've never really communicated with people who have been able even break into the Vita a little bit and B) I haven't either the skill or experience to do so myself, so there are quite likely things that are wrong in this post simply because there are things I don't know about the Vita.
Barring the ability to dump RAM from the device, it seems like what needs to be done is to dump the firmware so that a disassembly-level investigation can be run. However, there are likely several things that are preventing us from doing this really at all, currently. If I had to guess, I'd say that the firmware itself is inaccessible from userland and would need to be dumped in one of three ways: a kernel exploit (hooray catch-22s), dumping the flash chip itself, or extracting it from update blobs. However, update blobs are obviously encrypted, and the flash chip itself is likely also encrypted, and since we don't currently have a kernel exploit, we don't know the keys to use to decrypt either of these, much less the encryption algorithm used on them (although again, if I had to guess I'd say it's likely stock AES). In each case, we have the catch-22 of needing stuff that can only be gotten out via the kernel to get into the kernel, leaving us in sort of a difficult place.
A counterpoint here though is that if the flash chip is itself encrypted then there must be some sort of boot ROM that is able to decrypt it before dropping into the kernel loaded on it, so there might be hardware keys that can be recovered, possibly through decapping a ROM chip somewhere on the board.
It really makes me wonder how the first iPhone jailbreaks were able to recover enough information to actually glean how to break out of the app jail. A userland exploit in iOS is theoretically not able to see much outside of its jail. Can we recruit comex or something?