I do something similar to this with my blinks. Where upper blink is allowed to go from top to bottom. And when the eyelids meet, the lower blink gets clamped to the current value of the upper blink. I just use the clamp node for this. Nice and simple.
I am not measuring the geometry of the blink. I am measuring the 0 to 1 range of the blink attribute. Which is much simpler! For mGear, it is driving a blendshape on some curves, so you can read that value. When the upper blink is at 0.7, the lower blink is clamped to 0.3.
If you can find a way to simplify measuring your mouth, it might be a lot easier to drive some controllers. For example:
- Mouth open = -1.0
- Mouth closed = 0.0
- Mouth up = 1.0
I wrote a blog post about using the combinationShape node. It is the node that is used to make corrective shapes in the Shape Editor. But you are able to use the node by itself. So it doesn’t have to drive just blendshapes. It can drive any controller attributes you want. There might be some ideas for you here.
* (Note: To get my blinks to work like this, I had to make some custom edits to eye_rigger.py so that it makes separate curves and attributes for upper blink and lower blink.)