I just noticed that the links in my comments have end-of-sentence periods added, so they will not work. Here are the corrected links: Mimic manual: http://docs.daz3d.com/lib/exe/fetch.php/artzone/pub/software/mimic/mimic_pro_user_guide_071408.pdf Articles and video about JALI: http://www.dgp.toronto.edu/~elf/JALISIG16.pdf http://www.dgp.toronto.edu/~elf/jali.html https://www.youtube.com/watch?v=vniMsN53ZPI
I have written about this issue MANY TIMES IN THE FORUM. a) The audio scrubbing in iClone 6 is a God-send. Thank you, Reallusion. b) In iC6, I have noticed the "Ah" viseme tends to be at 100%, maybe every time, resulting in a sudden and jarring "jaw drop" that is way out of proportion with the rest of the speech. I noticed that when using good-quality recorded speech from a human female. c) I agree about the excess number of visemes. d) Sort-of off-topic, and a bit minor, but I've noticed the waveforms lag the actual audio by about a quarter-second or so. Nothing major, but I should write it up someday. Closing - This is a very vote-worthy topic. I'll have to review my list of 10 votes and see if I should move one of my votes over here.
I found some recent research that might inspire the development team: http://www.dgp.toronto.edu/~elf/JALISIG16.pdf. There is video that goes with it: https://www.youtube.com/watch?v=vniMsN53ZPI. And some more details here: http://www.dgp.toronto.edu/~elf/jali.html.
I have done quite a bit of lip sync with iclone 6. It is tedious and needs quite a bit of refinement. Most of the time I find that I am reducing the number of visemes and reusing the same ones just to get a somewhat believable lip movement. I think the key is to add more bone and morph control of the face based on the sound files. This is something that has been accomplished years ago by other software developers.
I don't why this doesn't get more votes, because the technology is there to do things better. Mimic, mentioned above, is a good example. I have used Mimic from the time that it was a Poser plugin and produced by a company called LipSync. I have written this many times in forum posts, but might as well repeat it here: one of the reasons Mimic works so well is that it uses a script of the dialog text to support the viseme assignment. To quote from the Mimic 3 manual: "The Text area of the Session Manager is where you load plain ASCII text (.txt on a Windows PC) files. Alternatively, you may click in the Text Display field and then type in your desired text. Your text must quote the sound file verbatim. This step is optional, but typically improves results. Mimic analyzes this text file (if present in conjunction with the sound file) to ensure maximum phoneme accuracy. It does this by reading each syllable, applying standard English pronunciation rules, and selecting the best available phoneme for the job." The manual is actually quite thorough and may serve is a source of inspiration: http://docs.daz3d.com/lib/exe/fetch.php/artzone/pub/software/mimic/mimic_pro_user_guide_071408.pdf. There is no shame in borrowing good ideas.