Home Website Youtube GitHub

Selection mask toggle

Rigging General
#1

Hi,

can someone help me create a toggle shortcut for toggling the selection mesh mask for the viewport?
image

I couldn’t find online a Phyton version of setObjectPickMask. This is what Maya calls when you click on it:

setObjectPickMask "Surface" false;
updateComponentSelectionMasks;
updateObjectSelectionMasks;
dR_selTypeChanged("");

Thanks!

#2

Hey there @Gui,

So I actually have never tried this before myself, but looking around a bit. The “selectionType” command is what you are looking for in this case.

What I did from your code snippet was search the Autodesk mel commands to see what it is doing under the hood. Take a look at this path or equivelent to where your maya is installed.

C:\Program Files\Autodesk\Maya2020\scripts\others\setObjectPickMask.mel

They seem to be doing their own implementation of how they are switching those components on and off.

Not a direct answer, but hope it helps you onto the right path!

All the best,
Jascha

2 Likes
#3

Actually, what I think you want is the selectType() command.
https://download.autodesk.com/us/maya/2011help/Commands/selectType.html

Here are some examples of how you could toggle it. “oppositeValue” tests the opposite by using the Python not keyword.

But in my last example, I toggle on or off polymesh, nurbsSurface, and subdiv all at once by using the not and any keywords. But that doesn’t completely disable the icon, which means some other surface type is still active. And “Surface” isn’t an option in this command. So I just don’t know what other flags also need to be turned off to make the icon completely turn off.

import pymel.core as pm
# Turn all ON
pm.selectType(allObjects=True)

# Toggle Joints
oppositeValue = not pm.selectType(joint=True, q=True)
pm.selectType(joint=oppositeValue)

# Toggle NurbsSurface
oppositeValue = not pm.selectType(nurbsSurface=True, q=True)
pm.selectType(nurbsSurface=oppositeValue)

# Toggle Poly Meshes
oppositeValue = not pm.selectType(polymesh=True, q=True)
pm.selectType(polymesh=oppositeValue)

# Toggle (ALMOST) all surface types:
areSurfacesOn = [
        pm.selectType(polymesh=True, q=True),
        pm.selectType(nurbsSurface=True, q=True),
        pm.selectType(subdiv=True, q=True)
        ]
oppositeValue = not any(areSurfacesOn)
pm.selectType(polymesh=oppositeValue, nurbsSurface=oppositeValue, subdiv=oppositeValue)

So instead, if you want to toggle ALL surfaces, you could either try to find the other flags it wants. Or you could use an ugly mashup of MEL and Python:

import pymel.core as pm
import maya.mel as mel

# Test if ANY of these 3 flags are on or not.
areSurfacesOn = [
        pm.selectType(polymesh=True, q=True),
        pm.selectType(nurbsSurface=True, q=True),
        pm.selectType(subdiv=True, q=True)
        ]
# Get the opposite value. If they are all True, it will be False.
oppositeValue = not any(areSurfacesOn)
# Get true for True and false for False, for MEL syntax.
lowerCaseBoolean = str(oppositeValue).lower()
mel.eval('setObjectPickMask "Surface" {};'.format(lowerCaseBoolean))
4 Likes
#4

This is great! Thanks @chrislesage
I honestly don’t mind the Python/Mel mashup, it’s just for me to put this on a Hotkey to use it while animating :slight_smile:
This is all very informative in general, I keep forgetting I can use mel.eval to run mel commands inside phyton. Also that oppositeValue commands might be useful for future toggles I might need - thanks heaps!

meanwhile I’ve modified to mask joint selection:

import pymel.core as pm
import maya.mel as mel

# Test joint flags are on or not.
areSurfacesOn = [pm.selectType(joint=True, q=True)]

# Get the opposite value. If they are all True, it will be False.
oppositeValue = not any(areSurfacesOn)

# Get true for True and false for False, for MEL syntax.
lowerCaseBoolean = str(oppositeValue).lower()
mel.eval('setObjectPickMask "Joint" {};'.format(lowerCaseBoolean))
#5

For joints, you don’t need the MEL. Because joints is only one type, you can just use the selectType() command directly. The only reason to use MEL is because there is no flag to turn all surface types on or off at once (as far as I can tell.)

import pymel.core as pm
# Toggle Joints
oppositeValue = not pm.selectType(joint=True, q=True)
pm.selectType(joint=oppositeValue)
#6

i like pm.mel over mel.eval for a slightly less ugly mash up

import pymel.core as pm

# Test if ANY of these 3 flags are on or not.
areSurfacesOn = [
        pm.selectType(polymesh=True, q=True),
        pm.selectType(nurbsSurface=True, q=True),
        pm.selectType(subdiv=True, q=True)
        ]
# Get the opposite value. If they are all True, it will be False.
oppositeValue = not any(areSurfacesOn)
pm.mel.setObjectPickMask("Surface", oppositeValue)
2 Likes