Layer Masks And The Use Of Luminosity Masks, A Tutorial

[blumesan]

LAYER MASKS AND THE USE OF LUMINOSITY MASKS, A TUTORIAL

First, let me state that this tutorial is not designed for PhotoShop tyros*. It is directed to those who have a passing acquaintance with layer masks, who have perhaps tried this technique a few times, but are still not quite confident of how, why and when to use a layer mask to confine and control the effects of adjustments. (*tyro: a perfectly fine synonym for newbie that has been in the language for at least 100 years. Just proves that neologisms are not always born of necessity.) The old hands who regularly use layer masks as part of their workflow can probably skip directly to the end where I discuss Luminosity masks and the actions that accompany this tutorial.

Second, I apologise to those who expect that tutorials should be replete with screen images and little arrows showing where to point your mouse. Unfortunately I don't have the patience to do it that way.

Some Basic Concepts:

• A layer mask is nothing more than a grayscale image. The default behavior is "white revals and black conceals". Translated this means that where the mask is white any adjustments made to that layer are fully implemented, and where the mask is black the adjustments have no effect. The gray areas of the mask allow a partial effect. What is really happening is the black portions of the mask cause the corresponding portion of the image to become transparent, and the gray portions semitransparent. In such areas you are seeing pixels from image layers lower down in the layer stack. (To visualize this in an image layer, just turn off the visibility of all the layers below the layer containing the mask.) Naturally any adjustments made to the layer will have no effect on transparent areas. The entire logic of using a layer mask is to control where and to what degree image adjustments are implemented.
• A layer mask always starts its life as a selection. There are innumerable ways to create a selection and I shall not even begin to elaborate on that topic. Once the selection is made you have two choices: 1) Immediately employ that selection to create a layer mask, or 2) Save the selection as a channel for later use (or re-use). If you have devoted some time and effort to creating the selection, the second method is highly recommended. It is simply accomplished: Select>Save Selection. Give it a name. When you look in your channels palette you will see the mask in all its black/white/gray glory. At this point it is just a mask, not yet a layer mask. To use it as a layer mask it must first be converted back to a selection. More on that later.
• This discussion will concern itself with only two types of layers: 1) Image layers (layers containing all or part of your image) and 2) Adjustment layers (layers added for the purpose of adjusting the image, such as Curves, Levels, etc.). I belabor the obvious here because, as we shall see, the method of adding a layer mask to each of these layer types is slightly different. I guess this is a good point to acknowledge that adjustment layers are not a necessity of life. Any adjustment for which an adjustment layer is available can also be performed directly on the image layer. This is generally considered poor practise and most experts recommend against this technique. However for those that insist in taking this route, a layer mask can still be valuable. While you can't go back and tweak the adjustment, you can still edit the layer mask. For certain adjustments, in particular all those employing the use of a filter, adjustment layers are not available and the adjustment must be performed directly on an image layer. (This may have changed in CS3.) In either case the use of a layer mask can be very effective in confining your adjustments to specific areas of the image.

Adding a Layer Mask:

• Generally the first step is to activate a selection. The marching ants should be visible. If you have just made a selection, you're ready to go. If you have stored the selection as a channel (and in the interval it has been deselected), or if the channel (mask) has been created by an action, you must load that channel as a selection: Select>Load Selection and choose the channel (mask) you wish to use.
• To add the layer mask to an adjustment layer: The selection must be activated before adding the adjustment layer itself. All adjustment layers come with a layer mask. If a selection is active while adding an adjustment layer, the selection will automatically be used as the layer mask. Choose either: Layer>New Adjustment Layer or simply click the icon (Create new fill or adjustment layer) at the bottom of the Layers Palette.
• To add the layer mask to an image layer: Again the selection must be active. Simply select the image layer to which the mask will be added, then Layer>Layer Mask>Reveal Selection or click the icon (Add layer mask) at the bottom of the Layers Palette. Right here is a good place to warn you about a mistake I make at least once with every image I edit. Immediately after adding a layer mask to an image layer you will note that the layer mask thumbnail itself is framed in white. This means that any adjustment you now make to that layer will be applied to the mask, not the image. Very frustrating if you next try to apply something like sharpening or blur to your image. Nothing happens (or at least not what you expect). To edit the image, not the mask, you must first click on the image thumbnail.
• Special case, the null mask: It is not always necessary to have a selection active when creating a layer mask. As already noted, every adjustment layer comes complete and unbidden with its own layer mask. You need do nothing. In this case (with no selection active) the mask will be all white (reveal all). A layer mask can be added to an image layer when no selection is active by choosing Layer>Layer Mask>Reveal All or again by clicking the icon at the bottom of the Layers Palette. In both cases the mask will be all white (really no mask at all) but this can be changed later (see Editing Layer Masks). To be complete I should mention here that choosing Layer>Layer Mask>Conceal All or Alt-clicking the icon at the bottom of the Layers Palette will produce an all black mask. This is sometimes a preferred starting point for later editing the layer mask.

When to Add a Layer Mask:

• You can add the mask either before or after making adjustments to the image. The mask will modify the adjustments, in exactly the same way, in either case. In the case of an adjustment layer, as already noted, the layer comes with a mask when you create it. If you desire to use a particular mask, it is much more practical to add the mask simultaneously with the creation of the adjustment layer as described above. Later that mask can be edited if desired, or even replaced altogether. In the case of an image layer mask, the circumstances will dictate whether it is best to add the mask before or after adjustments are made to the image. For example it is common to make global adjustment such as sharpening, noise reduction filters, or blurring first and subsequently adding a Reveal All or Conceal All layer mask which can then be edited to paint out or paint in the adjustment in selected parts of the image.

Editing a Layer Mask

• There are essentially two ways to edit a mask. First, if you have saved a selection as a channel, you can edit the channel itself before using it to to produce the layer mask. However the preferred and much more common method is to perform the edits directly on the layer mask itself after it has been created. Which is most efficient or useful will depend on what you are trying to achieve. The second method is preferred because it allows you to see, in real time, the effects on the image which result from editing the mask. (a bit more on this later). In both cases you have nearly the full range of PS editing tools at your disposal, but remember your edits will be confined to changing a grayscale image, thus making some parts darker and/or other parts lighter. The aim being to hide or expose more of the image to the adjustments which the mask controls.
• To edit a channel just go to the Channels Palette and select the desired channel. Of course in this case you will see only a grayscale image and apply your edits directly to it.
• To edit a layer mask make the appropriate layer active. If it is an adjustment layer you are ready to edit the layer mask. If its an image layer, first click on the layer mask thumbnail (a white frame will appear around it) so that you will be editing the layer mask and not the image itself. In both cases the image itself remains visible and any changes to the image which result from editing the layer mask are instantly apparent. To directly view the grayscale mask itself, Alt-Click on the mask thumbnail. You may continue editing with this view if it suits your purpose.
• As noted above, there are a variety of tools (confined to those that are effective on grayscale images) which can be used to edit a layer mask. For changing the luminosity of the dark and light areas (in order to increase or decrease the effect of the image adjustment) both the Levels and Curves adjustments are ideal. Remember these edits must be applied directly to the mask by using the menu choice: Image>Adjustments>. You can't use an adjustment layer to adjust a mask. (I only mention the obvious because, in my ignorance, I once tried it.) Another, and perhaps most frequently used method, is to paint on the mask using a black or white brush; usually, but not always, with a soft brush and low opacity. As noted above this technique is frequently used on an all black or all white mask to paint in or paint out adjustments previously made to the entire layer. I guess it can't be said too many times: "Black conceals and white reveals." There are lots of neat tricks to achieve special results when editing a mask, but that is a subject for another tutorial.

About Selections and an Introduction to Luminance Masks:

I said a few words about selections earlier, and there is no need for me to emphasize that creating them with the common tools (lassos, polygonal, magnetic or otherwise) and even with the magic wand or color range, is a pain in the derriere. Sometimes there is no choice, but even then such selections cause me grief in trying to get them to blend without obvious edge effects. So if there is an opportunity to create a selection that has essentially no edges as such, but still works, I leap at the chance. Selections based on the luminance of the image pixels offer such an opportunity. Pixel luminance does undergo considerable changes across the image but, even where there are sharp light-dark transitions, the luminance change at the individual pixel level is not nearly so abrupt as one might suspect. Thus masks (selections) based on luminance have a built-in type of feathering that is much more effective than what one can achieve by intentionally feathering (or blurring) the edge of other selections.
A simple and widely used variant of this method is to load the RGB Channel as a selection (Ctrl+Alt+~). This selection, used as a layer mask, will confine the effects of adjustment to the brighter parts of the image and protect the shadows. In designing the accompanying actions I have simply expanded on this idea.

• The Action "Select Highlights" will create 5 masks in your Channels Palette. Mask #1 is the same as one gets by loading the RGB channel as a selection. Masks #2 through #5 progressively narrow the selection (the revealed portion) to the brighter areas of the image and simultaneously conceal increasingly more of the shadows. E.g. using mask #5 as a layer mask will allow you to confine the adjustments to only the very bright highlights in the image.
• The Action "Select Shadows" will create 4 masks in your Channels Palette. Masks #1 through #4 progressively narrow the selection (the revealed portion) to the darker areas of the image while concealing increasingly more of the highlights. E.g. using mask #4 as a layer mask will allow you to confine the adjustment to only the deeper shadows in the image.
• Tip: If you usually save your edited images as a PSD file you can reduce the file size by deleting any unneeded channels. Right-Click the channel and choose delete.
• These masks are best suited for creating layer masks in adjustment layers (especially Curves, Levels, Color Balance, Hue/Saturation). They are also useful in image layers, particularly when employing different Blend Modes to create special effects. That is also a subject for another tutorial. Perhaps someone will come up with other ways to employ them in image layers. If so, be sure to let me know.
• The real advantage of these masks is that, even in the most extreme cases (Highlights #5 & Shadows #4) the transition between the revealed and the concealed portions of the mask are sufficiently gradual that no edge effect will ever be noted.
• Last, but not least, these luminosity masks are very amenable to further editing in order to refine their effect. Alt-Click on the mask thumbnail to bring up the grayscale image of the mask. Use either the Levels or Curves adjustment to edit the mask. You can make the bright areas still brighter and/or the dark areas still darker to further confine the adjustments to specific areas of the image. Naturally you can also go in the opposite direction to broaden the effects on your image. Alternatively you can keep the image in view as you make these edits to the mask. Just a simple click on the mask thumbnail (for image layers) will ensure that your edits are applied to the mask, and you can see the effects as you proceed.

Final Instructions:

• Download the Action (.atn) files from: Luminance Masks Action File
• Save the files whereever you usually save action files. Doesn't matter where as long as you remember where.
• Go to the flyout menu of the Actions Palette and select Load Actions. Navigate to where you saved the actions. Select the file "Luminance Masks, mrb.atn"
• Choose and run the action. Check your Channels Palette, the masks are there. Load one of the masks as a selection (instructions above).
• With the selection active, add an adjustment layer. Now you're on your own.

A final word (or two):

I can guarantee two things: 1.) The methods I describe above will work (if they don't please let me know and I will try to correct any errors), and 2.) There are at least a dozen different methods in PS to achieve the same results. Use whatever method best suits your workflow.

I will be happy to answer any questions directly related to the concepts and methods I have described. Going beyond that may well exceed my competence. I also welcome any comments and constructive criticism. If you have had the patience to read through this rather lengthy explanation, I hope it proves of some benefit to your work.

Mike.

[Paul Shields]

An excellent tutorial Mike, and thanks a lot for providing the Photoshop action files to accompany this.

[Worm324]

Excellent read Mike and thanks for the actions file!! Give this guy some Karma!!!


Jay

[dannyraphael]

Excellent writeup. One of the best I've seen on this subject.

Nice set of actions, too...

[Citybuff]

A very good tutorial, I will try it and let you know how I go. thanks

[Lutz]

thanks Mike,

the tutorial and the actions are agreat idea.

[narc60]

Mike,

Sorry newbie post, but when do you actually need to use this procedure? Any particular image or circumstance?

[blumesan]

Originally Posted by narc60

Mike,

Sorry newbie post, but when do you actually need to use this procedure? Any particular image or circumstance?

Use this tool whenever there are bright or dark areas in your image that you wish to adjust, without affecting the other areas. I'm sorry the tutorial was not written for the complete newbie, but rather for someone who already had some experience with masks. I suggest that you look for some simple tutorials on mask creation and use of masks, to bring you up to speed.

Cheers/Mike

[pam6203]

Am eagerly awaiting further PS tutorials.
Thanks.

Pam

[Goofup]

Good one!

[yendikeno]

Thank you -great tutorial!

[pop]

A fine tutorial.
Thank you so much for taking the time to do it.

[Aeronut]

Thanks for spending your time helping others understand these techniques a little more!

[firel7]

Thanks for an excellent tutorial and for the enclosed action.

[Lee4145]

I just wanted to bump this because it is a technique and an action I found really useful and still use more than I expected to.

Thanks again, Mike.

[Light Currents]

Excellent stuff, Mike. Thanks for your hard work. And well-produced too - in one large chunk that is easy and convenient to print or save for future reference.

Do readers have permission to print or save a copy to their harddrive as long as it is for personal reference (only) and not for redistribution?