Foliage Art for Forest Environments with Jonjo Hemmens

Hemlock Bark Material

The first material I created for this project was the Western Hemlock Bark. This material was seriously challenging for my skill level, but it provided a lot of opportunities to learn about shape generation and detailing. I spent about a month fumbling around with this material, finding my feet, and re-starting a couple of times.

Creating the fractured bark pieces accurate to the reference was very difficult and I still feel like it could be done better.

This was the primary reference for the material. I picked it because I liked the way that the bark plates split and peeled outwards. The sawn knots in the wood also have a really nice circular pattern. I thought they would be a lot of fun to attempt.

My material definitely ended up deviating quite a bit from the original reference: my knots are quite large and chunky, there is a lot more albedo contrast in the crevices between the plates. These changes were made to improve the look in-engine, but aside from that, I can see a lot of flaws in quality and detail that I didn’t achieve compared to the ref.

This would be the point where I would break down the graph, show screenshots of my work, and guide you through how I achieved my results. Unfortunately, my graph is an absolute disaster. The first half has some sense of structure, but as soon as I start detailing, it becomes a massive pile of un-readable spaghetti. I am really not proud of this one.

Returning to this material for this article, I tried for a few hours to untangle this mess, but there’s no saving it. I could not tell you really how it works anymore, and If I wanted to make a significant change to any of the details of this material, it would be a real struggle to find out where any changes happen.

I realized how bad this was too late to fix it and instead I made certain that my future graphs had proper structure so that you can follow the material as it develops and see where meaningful changes are being made. The graph for my dirt and needles material is much cleaner. I’ll talk about how I managed to downsize my graphs and my structure decisions for this a bit later.

Dirt and Needles Material

The base of any forest floor is usually scattered with lots of broken branches, needles, dead leaves, and pebbles, as well as a lot of plants. Although there’s a lot of nodes early on, I’m basically just trying to create a heightmap with enough interesting primary, and secondary forms to avoid any obvious landmarks when the material tiles across the landscape.

This noise gets pushed through a directional wrap which blends my noise map with a pre-made dirt map. This adds a lot of dirt details super quickly and blends nicely with the height information made previously. After that I sample some other noises to add some extra details, and I’m done.

With the dirt base is ready, I moved on to creating some of the assets to scatter around the material. The material is broken up into four different lanes to make it easier to see where a change is being made. The mask blending samples from the previously scattered assets create a black and white mask to feed into the next scatter to prevent assets from overlapping awkwardly.

Not only is it easier to understand, but it’s easier for me to now re-sample the assets I used in other graphs and materials which will save me time in the future.

Painting assets

After getting to a comfortable point with my materials in Substance 3D Designer, I decided to start on some plants. This was my first attempt at creating foliage assets, I had no idea how to create shrubs and plants properly. My head had been in hard-surface mode for the past year and I definitely needed a new perspective.

To start, I decided to jump headfirst into some tutorials created by excellent foliage artists. I referred to Peyton Varney’s Protégé Foliage Tutorial, and I also checked out Karen Stanley’s tutorial which follows similar workflows but has some excellent tips too. I’ve shared these tutorials more times than I can remember as it contains a lot of incredibly useful tips. I found that I prefer to assemble my atlas layout in my block-out stage before I start the high-poly stage and I usually texture 3D assets in Painter rather than Designer. Instead of repeating their tutorials back to you, I’ll cover the areas where I’ve deviated from their tutorials.

To work out what I wanted to model, I found a Native Plant Guide created for King County in Washington which was roughly where my scene was set. This website had all of the native plants found in the region, which I spent a good few hours sifting through. The website has tonnes of information about each plant as well as some shots for you to get a good idea of the type of plant. I ended up picking the plants based more on what I felt would be interesting or challenging to model over ones that would definitely be found in a forest.

The most interesting plants I ended up picking were the Rattlesnake Plantain, the Longipetala, and the Camas. Each of them had some really difficult modeling challenges and looked distinct.

Here are the block-out models for the Longipetala plant right before I brought it into ZBrush. I have the whole atlas blocked out so I know when my sculpting pass is complete, I can export and bake. If you’ve only modeled bespoke 3D assets before, this can be pretty confusing. Instead of modeling the whole plant in 3D, we make an atlas of parts that we can later unwrap onto geometry and model the plants after we have our textures ready.

As you can see in the screenshot, I’ve got my material colors set up early so I can create a transparency mask quickly. It also helps when you want to apply a material to a fiddly part of your mesh that would be frustrating to hand paint. I made the pollen tips on the flowers a yellow ID so that I would save time having to manually mask out each one. The blue strip on the side is a bit of a generic stem that I will use when assembling tube meshes to attach the flowers to the plants.

Peyton and Karen use Designer to texture their plants, but I’m a bit more capable with Painter, so I use that instead. Because I planned on creating a lot of plants with similar materials, I created a set of Smart Materials in Painter for both the petals and the stems/leaves. Both materials were generic by design. They included basic colors, roughness variation, micro height details, and some basic subsurface values. They were designed to be built upon and to save me from having to start from scratch each time. When starting a new Painter file, make sure you add the opacity and scattering channels before you bring in smart materials using them.

I then apply my smart materials and use the color selection mask in Painter to isolate each part. To save a bit of thinking, I always try to use the same colours for each element. For example, I always use the same pink hue for the opacity mask.
Rather than break down every material I made, I would push people towards the Painter Tutorials tutorials of both Jason Ord and Rick Greeve. Despite the fact they both cover hard surface assets, they cover a lot of fundamental skills in Substance Painter that I use regularly to get solid results with any asset you decide to texture.

Pushing Your Texturing Further by Jason Ord covers every part of the texturing process from how to identify good reference all the way down to setting up your own base materials and building up all the nitty-gritty details.

Creating and Texturing with Custom Stencils by Rick Greeve gives you a great idea of how important hand painting is and how you can go about creating your own stencils. When I’m happy with my textures, I’ll export and bring them into Maya and start cutting the atlas out into individual parts. Assembling some basic pieces first makes life a lot easier. With the Longipetala, I had a flat circular base on which I would layer the leaf clusters on top to give the plant some volume. If I plan on bending or deforming a mesh, I’ll add some additional edge loops, then remove any that don’t significantly improve the silhouette or shape of the plant when optimizing.

For the stems of the flowers, I created a triangular tube. I found that these looked far better than flat planes from all angles, and were only marginally more expensive. This variation of the plant is quite costly from a game development perspective at 912 tris, making it one of the most expensive ground plants in the scene. This was by far the most detailed plant in the scene and it’s used quite rarely in specific spots, unlike some of the other plants like the clovers which are used densely and almost everywhere.

Here are some Marmoset renders showing the plant off in all its glory. Although there is some clipping in places, it’s almost impossible to avoid this (especially when you add wind effects in-Engine). I was pretty happy with these in the end as they look quite close to their real-world counterpart, and they introduced some much-needed color variety into the scene.

Shot of the Longipetala in Marmoset

Shot of the Longipetala amongst the other foliage

Terrain Sculpting Process

I began by using large brushes, sculpting my large forms. I decided to have the scene to have one large hill, then pathways leading from the base of the hill, all the way to the top, with waterfalls and rivers to break up the middle of the scene.

With my primary forms in place, I paint my materials onto the landscape. I use it to decide where I would place my set dressing elements before I started sculpting my smaller details. I used the river rocks to define where my rivers would go, the mud for the paths, and the moss for dense patches of foliage. Wherever there are really steep height transitions, I usually use rocks to form a cliff face. Adding cliffs and rock clusters really help add structure to the scene, treat them like you would use walls in an interior.

When I set dress, I usually go from big, to medium, to small. I start with the largest assets such as boulders, mountain rocks, and trees. Getting these in early saves you time and energy by either blocking off spaces and giving you a smaller space to set dress with your smaller details.

Once I’m happy with my larger assets, I’ll scatter my foliage in the mossy areas, add some embankment meshes to the pathways, then hand place some rocks and tree stumps to make some areas of interest.

By being creative with my camera angles and set dressing, I was able to make the scene feel much larger than it actually is. As you can see in this top-down shot, most of the static shots take place in a pretty small part of the scene. The areas in red here aren’t seen in any of the renders because they’re occluded or the camera just never points in that direction.
Experimenting with compositions and camera angles early on helped me save so much time, and I didn’t have to dress a massive landscape in the end, just the areas that were most important.

Screen Space Global Illumination

One of the new beta features in Unreal Engine 4.26 is the real-time SSGI (Screen Space Global Illumination) solution. Global illumination simulates lighting interactions with geometry and material surfaces. The scene lights will bounce between surfaces, sampling colors from the light and the bounce surface to create more realistic lighting. Here is an example of it on and off in my project to better illustrate what it does.

As you can see, the SSGI adds a lot of bounce light in the crevices. It really helps to remove some of the darker values and softens the values out nicely. The values I’ve used here are the highest I can set, so in some places, it gets a little too intense.

One of the biggest upsides of using this is that it is super cheap to run, and provides pretty good bounces across your whole scene. The downsides are that the global illumination seems to get less intense around the edges of the frame, creating a vignette effect. I also found that I was relying on the SSGI to raise the values of my shadows, but on the borders, I was getting really dark values.

During my cinematic, I noticed that as the scene moved past certain objects, you can see some pretty severe flickering that doesn’t look correct at all.

And finally, I noticed that it could flatten the normals of some assets depending on how intense the SSGI was on that asset. On this rock, you can see that some of the heavier cracks almost disappear.

The tool is in its beta stages and I felt like it was still worth using for my project, despite some of the issues I ran into. To enable the SSGI for your own project, you only need to open up your project settings and search for it, then tick the checkbox.

Once it’s enabled, you can change the intensity with a post-process volume by modifying the Indirect Lighting Intensity value.

If you’re using Unreal Engine 5 or 4.27, you have other options. Unreal Engine 5 uses Lumen which comes with fantastic global illumination by default, and Unreal 4.27 has a plug-in for NVidia’s RTXGI tool This feature might only be useful for those still working on 4.26 or for those looking for a low-performance cost solution to GI.

Setting up Lighting Sub-Levels

Lighting the forest was particularly difficult because of many different camera angles, assets getting in the way. Once I was happy with the primary forms of the forest and I wasn’t going to make any major layout changes, I started introducing sub-levels that would have specific models and their own lighting set up for particularly tricky shots. Sub-levels are just like regular levels but can be toggled on or off in your current level. They’re also super handy if you’re working on a collaborative project.

Some of my sublevels for my cinematic shots

As I mentioned previously, the forest isn’t particularly large, and to save myself some time it was a lot easier just to build some of the shots with sub-levels. This meant that I could add a dozen or so large trees to a shot without blocking the sun for half the level. I could also have a different direction for the directional light in whichever shots I wanted. If you intend on using sub-levels and you have a cinematic, make sure you add a level visibility to your sequence, or the screen will go black.

Atmospheric Perspective

Atmospheric perspective is the effect the atmosphere has on the appearance of an object as viewed from a distance. I learned about this phenomenon through James Gurney’s book, Colour and Light, and wanted to achieve the same results. He describes it as; “The bold colors of the foreground gradually transform until they match the sky.”

To simulate this effect in UE4, I tried to create a post-process material that would change the color of the foreground and the background based on scene depth. Through some research online, I found that a good material for this already existed in Quixel’s Medieval Game Environment Project.

The material has a color value for both the foreground and the background. Here are some examples just to demonstrate how it works in my scene.

On the left, I’ve got the scene without any of the atmospheric perspective enabled, and my settings for the shot. You can see that my settings push the blues in the background without adding any blues to the foreground. I used some more extreme examples on the right to illustrate how it’s working. Although these are completely impractical they do a good job of showing the transition. Even if you’re not interested in using atmospheric perspective, the ability to manipulate color hue in the foreground and background is a really powerful tool.

Simulating a Large Environment (creating a video)

For the cinematic, I wanted to mimic some of the forest drone flythrough videos I’d seen on YouTube. They usually have simple, slow-moving camera movements through the forest, showing off some beautiful scenery. The simple camera movement was an excellent choice considering I’m a cinematics noob. I’m not an expert, but I picked up some useful tips that could help if you’re trying to achieve similar results.

Particle and Wind Delay

Something I noticed on my early renders was that the particle and wind systems weren’t moving until very late into the cinematic. This was because these systems need time to start when you start rendering or playing the scene.

To make sure these things kick off properly for your cinematic, you can add an engine warmup count which starts all of these processes without rendering out any frames. I added a 600 frame delay, which is probably a bit over the top, but with 4K renders, you really want to make sure your wind and particles aren’t still booting up.

The movie render queue is a really fantastic tool for cinematic rendering and you can do a lot of interesting things with it to enhance your renders. I’m going to talk about Will Faucher’s channel later, but here are a couple of videos of his to help you understand the tool and what it can do.

https://www.youtube.com/watch?v=FxvF3zncClA

https://www.youtube.com/watch?v=2U1wP8sJgfU

Subtle Camera Shake

In some of my shots, I use just a little bit of camera shake which makes the camera shots feel a bit more organic. Adding things like this to digital work help push things closer to realism, which is really hard to achieve. Here’s a great tutorial that covers how to set it up for yourself. Personally, I prefer really subtle camera shake, so I keep the values low.

If you want to know more about camera shake / cinematic post-processing in general, I recommend a series of blog posts done by Alex Tucker; There are some great tips and breakdowns here to help people understand how to control different types of post effects for their scenes.

Hiding Assets for Certain Shots

If you have some troublesome assets that you didn’t quite get sorted into the right sublevels, or an asset is just being difficult for this one particular shot, you can hide individual assets just for your shot.

Here I’ve hidden some of the taller trees in the shot to better shape the background. I simply dragged the asset from the outliner into the sequencer, added an “actor hidden in-game” track. From there you can keyframe whether the actor is visible or not for the shot.

You won’t be able to hide foliage-painted assets, as the foliage painter adds all of those assets into an instanced foliage actor. I ended up just replacing some of my painted trees with hand-placed ones so I could hide them.

Will Faucher

One of the most valuable resources I came across during the project was the videos by Will Faucher. He has a lot of learning content focused on Unreal Engine and Da Vinci Resolve. In his videos, there are countless useful tips, rendering tweaks, and breakdowns to help you understand how he pushes his cinematics further. I highly recommend his content for anyone looking to do cinematics in UE4.

Choosing when to finish a project

Deciding when to finish a project can be difficult, especially with large and complex projects. I’ve found that when the changes you’re making are becoming less meaningful and more subjective and opinionated, it’s probably time to set the project down and wrap up. You’ve hopefully learned from your project and it’s time for you to get some final polish done and wrap up.