Baking I

Texture baking is a step in our 3D pipeline that we use to transfer details and information from a source mesh (usually our highpoly) to a target mesh with UVs (our lowpoly or final in-game model, also called base mesh). To achieve this, our software or tool of choice for baking will cast rays from either the lowpoly mesh or a custom cage towards our highpoly model, and when those rays hit they will record the surface detail information from that detailed mesh and transfer it to multiple texture map types.

Don’t worry if you’re hearing about some of these terms for the first time, we will expand on each of these topics later in this lesson.

‍

The goal of this lesson is to give you an understanding of the whole process of baking and why it is so important, not just to get our normal maps but also to see what other information maps we can get from our bakes and how we can use them to help in our texturing process. For this, we will go over how to prepare your meshes for the bake to work properly and how you must organize and name your mesh components. We’ll also see the settings and tools you can use inside the baking software (Substance Painter and Marmoset Toolbag for this lesson) for best results, and how to avoid/solve common problems when baking.

The focus of this lesson will be on assets with ad-hoc (unique) texturing, since assets using tileables won’t really go into an individual baking process per mesh, but the knowledge that you will gain here can also be used to create your trimsheets or even tileables, since those can also have a bake to benefit in the same way as uniquely textured assets do.

There’s few things more satisfying to me than a clean bake, and it’s a very exciting moment when you can already see all the detail, shapes and volume of your mesh taking form and how well they will interact with lighting. A well done baking stage will set you up for success when going into the final texturing of your asset, which is the favourite part of a lot of artists, so let’s prepare ourselves to put the attention that this topic deserves, and let’s get started with this lesson!

BAKING THEORY

As we already mentioned briefly before, texture baking is the process of transfering details from one 3D model (our source, usually a highpoly) and save them as texture information, so we can apply those maps to our target and final model (lowpoly). We usually do this to be able to save polycount in our final model by having details just in our normal map and also because we can enhance our texturing workflow by using information related to the mesh geometry. Texture baking can be used to export different types of maps and we’ll go over that later in this lesson.

For this process, our baking tool starts a certain distance out from the model (our lowpoly), and casts rays inwards towards another model (our highpoly). This process usually involves a cage that tells the baker when to start and how far to extend in searching for that detail (we’ll expand on this in a minute). When a ray intersects with the highpoly, it records the surface detail and saves that information in a texture map, using the UV coordinates of our lowpoly model.

Cages

Cages can be smoothed or hardened (think about this in the same fashion as your meshes with smoothing groups and hard edges), and this links to our next topic, average and split normals.

‍

You can see a visual representation of a cage here. Essentially a copy of our lowpoly, extending out to capture all details in our highpoly and transfer them to our baked maps.

Average & Split Normals

Average normals is a way of calculating the direction of the vertex normals, doing an average (heh!) or middle point of the original directions of the vertices in your mesh (split normals) . You usually want the average normals option enabled, since this calculation will provide a better solution for baking your edges and bevels, but you may have other problems like skewed details on flat surfaces. If disabled, it will calculate the original normals (split normals) of your vertices, which can be beneficial to avoid distorted details but will usually result in an ugly seam in your edges.

You could also translate this to soft edges (average) and hard edges (split), but have into account that the geometry smoothing on your meshes can be set up in a way and then calculated in a different way in your baker.

Normal Format and Tangent Space Sync

Image from Marmoset Toolbag documentation

HIGHPOLY MODEL

Your highpoly model will be the source of all the detail in your baked maps, and the goal is that your lowpoly with bakes applied looks the same as your highpoly model. You will obviously notice some differences due having less geometry in your silhouettes/shapes, and less depth on details that no longer have geometry to support them and rely entirely in the normal map, but essentially everything you see in your highpoly model should carry over to the LP+bakes.

While this lesson is not about highpoly modelling, there’s some essential things that you will need to have into account that will directly affect your bakes, so I recommend to put special attention to things like having clean shading across your entire model, create bevels that are wide enough to be visible even from a distance (but be careful to not take it too far, you want to avoid your models looking soft and melty, it’s all about finding the right balance), and to be sure that your highpoly model shows exactly how the asset would be constructed in real life (separation from multiple components, welded pieces, no intersections or clipping, bolts that keep everything together, etc).

For a more in depth look about this topic, check out our lesson about highpoly modelling: {{ "MODELLING I" }}

Floating Geometry

Floating geometry are details that you can model in your highpoly that don't need to be connected/sewn to the main surface, but placed right on top of It. Since these details will only be in your highpoly and not present in the lowpoly, you just need them to transfer properly to your baked normal map. Doing this can give you the same exact amount of detail while saving time on modelling complex shapes and flexibility when iterating on these details.

‍

These highlighted details on this M16 model are floating geometry, placed on top and not present in the lowpoly. This allows us to easily bake some detail without having to cut it into the main shape, which would take more time and will be harder to have good shading.

Depth vs Angle

You need to have into account that with baking, the info that you are getting is not depth, but info that represents how the light changes direction and interacts with the surface of your object. That's why when modelling the details in your highpoly that are supposed to go in our out of the main surface model, you will benefit much more from giving a bit of an angle/slope that catches that lighting info than from making the details have more or less depth, since the info will be basically the same no matter how much you extrude it or inset it.

Here’s an example where having an angle on your detail will bake much nicer than just a straight inset. No matter how much you push that detail back, the bakes won’t improve to show that depth.

Assigning ID

When your highpoly is ready, you will benefit from assigning different materials to each of the different parts of your model to be able to bake a material ID map that can be used for múltiple purposes, like easier masking for example. You can separate your highpoly however you see fit depending on what you need, but some common separations are per different material or per component of the asset.

‍

Here we’re assigning a different material to our highpoly to create an ID map that we can bake and use for easy masking

LOWPOLY MODEL

The lowpoly model will be the final asset that will receive the info from the baking (coming from the highpoly), the one you’ll make textures for (usually) and what will be imported into your engine.

For the baking process to work, you will need proper UVs and correct shading with no artifacts. While the baked normal map can hide some imperfections and transform the look of the lowpoly, It won't solve any major issues, so you want to put special attention to your smoothing groups and hard edges.

While you want your model to be as optimized as possible, you need the lowpoly shapes to be as close as the ones in your highpoly to avoid any errors and achieve clean bakes, so put special attention to your silhouette and make sure that you have enough geometry to avoid a faceted look. The baked maps can only go so far and while they allow you to optimize a lot, they still need a solid base to work.

For a more in-depth look about this topic, check out our lesson about lowpoly modelling: ****  {{ "MODELLING I" }}

Seams, hard edges and soft edges

The basics here to apply proper shading to your models is that hard edges are sharp transitions from one shape or direction into another (e.g. a sharp corner, a 90º angle…) while soft edges will represent smooth transitions (e.g. the sides of a cylinder, a sphere). You can control this in your modelling package of choice to ensure that your asset looks how you want it to look. Overall, you want your lowpoly model to have proper shading so it can work on it’s own and use the normal map to enhance your shapes and details.

Seams represent the cuts in our model and boundaries of our UV islands (you can learn more about this here:  {{ "UNWRAPPING I" }}

When it comes to baking, you must make sure that anytime you assign a hard edge to your model, you also need to place a UV seam in that edge, otherwise you will have baking errors on that edge. There’s no workaround to this, but luckily this will be easy enough since most modelling packages offer a quick way to select hard edges/different smoothing groups for you to assign seams to them in no time. This doesn’t need to work the other way tho, not every seam needs to be a hard edge and you can use this to your advantadge to optimize your UV packing (e.g. if you’re modelling an hexagonal nut, you can represent those sharp transitions in your highpoly but have the 6 sides of your lowpoly model fully smooth and with just 1 UV seam, the baking will take care of making it look how it should)

‍

Overlapping UVs

When unwrapping your Lowpoly model, you are most likely to have mirrored and overlapping UV islands to optimize your texturing. When preparing your mesh for baking it's important that the overlapping parts in your packing are outside of the packing box (usually you offset them by 1 in any direction) to avoid baking errors.

‍

Everything mirrored or overlapped needs to be outside the packing box for baking to avoid errors, afterwards we can put the islands back in place.

EXPORTING FOR BAKING

There are some good practices to keep your workflow as solid as possible, error-free and as combinient as possible for you to have an easier time with baking your meshes.

Correct naming

It’s important that all the components of your mesh are named correctly for baking. First, you will have a better understanding of what each component in your scene is, and this will also allow other people working with your files to clearly understand your work. But on top of organization reasons, naming your components correctly will also allow the baker to identify which parts of your lowpoly match with your highpoly, enabling some options like “Match by mesh name”, which allows you to bake each of your components individually to each other, so you can avoid intersection or AO errors in certain places.

To do this, your highpoly and lowpoly element’s names will need to match each other, with the difference of the highpoly components ending with the suffix _high and the lowpoly components ending with the suffix _low.

Let’s say you’re modelling a door and your elements are a frame, a door and a handle, your higpoly names will be: frame_high, door_high and handle_high

Lowpoly names will be: frame_low, door_low and handle_low

With correct naming we can control our bakes much better and avoid unwanted AO or intersections in our bakes.

Exploding your mesh

Exploding your mesh means to physically separate each part of your mesh, offsetting them from their original position so they are not close to each other, in order to control how AO and intersections are baked. This is usually done when you don’t want pieces of your mesh casting AO to others, for example in meshes that need to animate and therefore you can’t have baked AO in the moving parts.

This is less common nowadays thanks to the Match by mesh name option mentioned before, allowing you to control how AO affect other parts, but sometimes you’d want some parts affecting each other, and some others that don’t get affected at all. You can then explode those parts of your mesh that you want to bake separately to allow for this extra control.

‍

Triangulating your mesh

I recommend that before exporting your lowpoly mesh for baking, you triangulate it. This is because each software triangulates the mesh in a different way, and while in some DCCs you can have control of it, in some others you don’t, and this asynchrony can lead to some issues with shading. What you see working perfectly fine in a software may have an issue in another, and while having good topology will make sure that your model works well mostly everywhere, you want to eliminate the chance of having small, uncontrollable errors.

I recommend that you keep this triangulation non-destructive, for example using modifiers if your modelling software allows (most of them will), or at least by saving a copy of the non-triangulated version, in case you need to go back to it later.

FBX vs OBJ

Some people have doubts about which format to use when it comes to export their meshes for baking. This (as a lot of things in your workflow) will depend on your needs of course, but when it comes to games it's almost guaranteed that you will end up using FBX for your final mesh, so I recommend that your lowpoly for baking is also an FBX (or even the same as your final mesh).

For highpolies it can be a bit more flexible (since it won't be used in the end product, just to get info for the baked) and comes down to needs and preferences. FBX works totally fine and keeps your all files consistent, while using OBJ can have it's advantadges, for example if you prefer exporting with that format directly from ZBrush. Sometimes Marmoset will have an easier time loading OBJs too when it comes to very huge and high polycount meshes, so this can also be a thing to have into account.

BAKING SOFTWARE SETTINGS AND TOOLS

This section will be a bit more straight to the point and just to give you a basic idea of the tools you have available in the most common bakers. You need to have into account that all settings are there for a reason, so there’s not just one way of doing the process. This will depend on a variety of factors, like the needs of your project, what mesh you are creating and the workflow you are using, target engines and even how much your computer can handle. Feel free to experiment and divert from these tips to find the solution that best suits your need.

Importing

In Substance Painter this is pretty straightforward, you will import your lowpoly/final mesh when creating a new project and after that, you can go to the baking section to import your highpoly if you need one.

For Marmoset, instead of importing the meshes straight away, I recommend that you make sure to create a new Bake Project inside your scene, and use the Quick Loader tool in there to import both your lowpoly and highpoly. Given that you already have a good naming convetion for both your low and highpoly meshes (we already mentioned this before in the lesson), Marmoset will read that and assign each component of your mesh to the appropraite folder, allowing you to control the cage offset and skew painting individually, and do your bakes matching by mesh name. You can also have the Auto-Reload option enabled, so if you need to re-export your mesh from your modelling software, Marmoset will update the file automatically, saving you time on re-importing assets inside Marmoset.

‍

Match by Mesh Name

We already covered this topic when taking about naming of your exported meshes, but to recap, you can enable this option in both Substance Painter and Marmoset Toolbag so you can bake each of your components individually to each other, avoiding unwanted info, intersections or AO errors in certain places. In Marmoset you can control this by enabling and disabling the “Ignore Groups” option on the individual map baking options.

‍

Skew Painting

‍

GIF from Marmoset Toolbag documentation: “Projection Tools”

Common Settings

Most bakers have the same options and settings for baking, and usually you want to do your bakes with as much quality as possible for the best results and most detailed maps, carrying over to your texturing since a lot of your info will be based on the bakes. However, if your machine is not capable of baking in the highest quality settings or it takes too long for you, you can adjust and reduce quality settings slowly, to find the sweet spot for you and your project needs.

Try to bake at the maximum resolution possible always (usually 4k 8k for some), even if your final exported textures are going to be of a lower resolution, you will benefit of having your bakes at the highest resolution if you can. You want to crank up your antialiasing samples too, going to x64 if you can. Some maps like AO, Curvature or Thickness will let you choose the amount of casted rays for the bake, so try to increase that to the maximum amount that your baking software offers and your computer can handle. If you can select your bit/channel format (like in Marmoset), I find that 16 bits/channel usually works best for bakes, since some generators and smart masks in Substance Painter will need that format to work well (again, this has nothing to do with your final exported textures and can be different, just talking baked maps here). You will usually want to add some padding in the baking too, default settings usually work quite well.

There’s a lot more options on each baker, but with what I mentioned before is usually the standard for most bakes. You will have to play around with the rest of the options to find what works best for each asset and project, and with time you will be able to know in advance what will work, reducing your testing times for each bakes, so as with everything, go bake some stuff if to get better results in baking!

USEFUL BAKED MAPS

In this section I want to briefly cover some of the most useful maps we can get from our baking process to get the details we added in our highpoly model, and some of their common uses in texturing or shading in engines.

• Normal map: This is the most common of the maps, and probably one of the most impactful. Normal maps contain info that catches the light trying to represent how real geometry would behave, such as grooves, bumps, bolts, etc. You would usually use this on it’s own and then add more info or get advantadge of it with your texturing in Substance Painter.
• Ambient Occlusion: Another pretty common and useful map. This map represents how exposed to ambient lighting each point of your object is. Crevasses, concave areas and interior of the objects will be much darker, while more exposed areas will be lighter, simulating “shadow and lighting”. This map alone will give a lot of volume and depth to your mesh, but you can also use this info to create masks for your texturing, simulating how dirt would build-up in your assets, add rust in places where water would collect, etc.
• Curvature: This map stores the info about the convex and concave parts of your mesh (pits & peaks). It’s really useful to create masks driven from it’s info, for example to create worn edges or other kinds of damage on the more exposed parts. It’s also common for more stylized texturing approaches to use it for edge highlights.
• ID Map: ID maps are essentially masks that have a variety of colors (or different B&W values if you choose to have it as a greyscale map) that contains info to quickly identify or mask different parts based on how you choose to separate them. You can separate all the components of your mesh from each other, you can group them by the material they’re made of, etc… This is commonly used to mask out different parts of your object quickly, or even to assign different materials in engine, amongst other things.
• Position: Baking the position map calculates the location of the highpoly model and creates a map with that info depending on the parameters you use (one axis, various axes…). This is very useful for some generators, and also to create masks for different effects, like dirt that builds from the ground up and it’s not present on the top of the asset, or snow/moss that would usually build up on the top areas of the asset.
• Thickness: A thickness map is very similar to what an Ambient Occlusion map would be, but the baker casts the rays from the inside out, identifying which parts of your model are thinner (black) and which ones are thicker (white). You can use this info to create some masks for your textures, but one of the most common uses is to create a Sub Surface Scattering map from it, which a lot of people start by inverting using an inverted thickness map and refining on top of it.
• Opacity: This map will indicate which parts of your object are opaque (white) and which ones are transparent (black). It can also work in grey values if you want to make something translucent, as long as your shader supports it. If you want to bake this from your highpoly, you just need to make sure that you have your geometry in the highpoly model as it would be in real life, and  bake it down to whatever you need in the lowpoly (usually a flat piece of geometry).

COMMON ISSUES

Most of the common issues have already been mentioned previously in this lesson, and we’ve gone through good practices to avoid them in the highpoly and lowpoly modelling stages, but here’s a handy recap of common issues so you can easily identify them and know how they look, why they happen and tips on how to solve them for a clean baking process.

Not enough geometry to support silhouette

This is a common error that happens when your lowpoly mesh doesn’t have enough geometry to support the shapes of your highpoly, The difference in geometry causes a difference in silhouette too big for the baking process to take care of it completely, and this will usually look like black artifacts across your silhouette, and a very low poly look. This is solved by just adding more edge loops/geometry on the areas where you see this happening.

A trick some people use to avoid this error but keep the geometry of the final mesh as low as possible, is to use an additional “baking mesh” with a bit more polycount than the final mesh, and use that one for baking, but then apply the baked maps to that final mesh with a lower polycount. This may work in some cases and will alleviate the problem, but won’t get rid of it completely and you’ll see it come back again when using your final lowpoly mesh (not as much as if you were not using that secondary baking mesh, but you’ll still see it). Geometry these days is not as much of a trouble for performance as other things (like texture usage for example), so in my opinion is worth adding that extra geometry to your meshes and just use that, as long as you keep it reasonable and justified, a few more polies won’t really make that much of a difference.

‍

This is the kind of error that you want to avoid, caused by very few edges in our lowpoly. In this case this wasn’t visible due to the camera view (isometric), but if you were making this for a portfolio piece or a first person game, you could easily solve it by just adding more edges to the silhouette which would get rid of those artifacts.

Skewing

We say we have skewing on our bakes when the details in your highpoly appear distorted/deformed when baked into our lowpoly. A good example of this are bolts baked into flat areas of your lowpoly, that appear like a deformed oval instead of respecting their original shape from your highpoly model. A reason for this is having a large flat surface without any loop cuts in it, and also a product of what method we are using to calculate our ray directions.

In Substance Painter, you may try to solve this either by adding some supporting geometry on those large flat areas, or by disabling “Average Normals” on your bake settings. However, disabling this options will result in other errors (seams) in the hard edges of your model, so if you choose this route you will need to bake two versions (one with Average Normals enabled, and one with it disabled) and combine both maps (either in Photoshop or Substance Painter) to get the correct section of it and end up with no errors. Follow this video for a step by step process on it: https://www.youtube.com/watch?v=jV_kuxeD8q4&t=284s

However, as we explained before on this lesson, Marmoset Toolbag has a much better way of dealing with skewing errors without having to do that much trickery. You can simply use the Paint Skew tool, which will give you a brush that you can use to paint over your skewed areas, and that will fix the error. I gave you the options for both softwares here, but I cannot recommend enough that you check out Marmoset Toolbag to do all your baking on it, since you will have a much easier time and your baking process will be much cleaner and faster once you get used to it.

‍

Image from Beyond Extent - Weekly Tips “Baking Troubleshooting 2”

Seams

When having this error, you will notice undesired cuts and discontinuity in your bakes instead of having a smooth surface. Some seams will be there just because of the UV cuts, but those can be hidden and won’t be too noticeable, but in this section I’m talking about seams that present a bigger issue and that can be totally fixed.

The main reason for this error will be a result of having a hard edge that doesn’t match to a UV seam, so you need to make sure that every hard edge in your model also has a UV seam assigned to it.

Another reason for this may be because of the settings of your baker, specially when using Substance Painter. When using this software, make sure the “Average Normals” setting is enabled, and this will make your bevels bake nicely to single edges of your model.

‍

Image from Beyond Extent - Weekly Tips “Baking Troubleshooting 1”

Artifacts (overlapping)

We already covered this in the Lowpoly section, but since we’re talking about common errors I may aswell add it here too. When you are going to bake a mesh, you can’t have overlaps inside the 0-1 UV space or you will end up with artifacts in your bakes. This doesn’t mean you can’t have overlapped or mirrored UV islands, it just means that before baking you need to offset all those mirrors/overlaps by a quantity (1 in any direction is usually fast and easy to revert back) so all those islands are outside of the packing box, and the baking software doesn’t have trouble with them. Move them back to their original place after the baking is done.

Baked info in undesired places

This is not an error per se, but a common issue or undesired result when baking. This happens when you end up with info like AO in places that should not have that information visible, like for example on moving parts or elements that you mirrored/overlapped but have different objects around it providing info.

You can fix this with a bit of planning around your models, and either choosing the “Match by name” option if baking in Substance Painter or working with bake groups in Marmoset, or exploding (moving away) the parts that you don’t want to be affected in your bake (both methods were alredy covered before on this lesson).

‍

Image from Marmoset Toolbag documentation

BAKING A TEXTURED HIGHPOLY TO A LOWPOLY

Most of the times you will do your textures on a Lowpoly, which will be the final version that goes in the engine, but there's some cases where your textures will be done on a higher polycount model for some reason, and you will need to transfer (bake) those textures to a lower poly model that may have different UVs and where a regular LOD or polycount reduction just doesn't cut It.

The cases can be for example working on two versions of a prop or weapon, where your need a cinematic model with higher polycount or even different sizes (animation needs), and then you need a lower poly versión for the game itself. Another case can be if you need to create both a Nanite and a regular LOD0 versión of your asset, maybe for market place content or because you’re aiming for very different platforms.

This can be done using Marmoset Toolbag: you just need to import your higher and lower poly versions same as in a regular bake, create a material with your textures to assign it to your source (highpoly) model, select the texture maps you want to bake (albedo, rougness, etc), and Marmoset will transfer the textures down to the new mesh (lowpoly) with new UVs. Pretty neat if you ask me!

‍