Tutorials

# Live Render Preview with LONET v2 [00:00:00] This is an update of our Live Render Preview tutorial, now that we've switched to the LONET2 protocol for our real time tracking output. Live Render Preview is a new feature that lets us drive an external rendering engine, in this case Unreal, with the real time camera tracking data from Jetset and the live video from a Cine camera. We now use the LONET2 standard for real time camera tracking data to send the position and orientation and the Cine Lens properties of Jetset Cine to configure Unreal's camera to have a matching field of view. We use an inexpensive Elgato HDMI capture dongle to bring the live video into the PC, and then we can use the Realtime Keyers in Unreal's Composure tool to key out the blue or green screen elements in the live action video. I'll put the link to the Elgato HDMI capture dongle in the links below. The same technique should work with most video capture devices compatible with Unreal. This will give us a real time rendered lighting preview of the live composite, including the rendered background in Unreal. I want to thank Rajashakar Techy for his [00:01:00] Composure tutorial that inspired this feature. I wanted our users to get a clean result like his setup without having to click their way through the Composure interface to set everything up. I'll put the link to that tutorial below. It's important to note that the live action and CG elements in this composite won't be exactly synchronized, as we can't yet genlock the engine to the incoming HDMI signal, but it's a good reference for lighting the live action scene to fit with the CGI scene. This tutorial will also serve as a building block for the additional step of adding in video captured through a Blackmagic or AJA capture board. And that will enable synchronization. If we capture video through those boards, we can include timecode in the signal. And that enables us to use the Unreal Timed data monitor to correctly synchronize the live action and the CGI composite. ## 24+ fps Required It's also very important to note a quirk of Unreal, which is that the media player element we're using to read the Elgato HDMI dongle won't accept any video frame rates under 24 frames per second. This means that a real time 23. 98 HDMI [00:02:00] output, which is extremely common in the US, won't work. You'll need to change the frame rate of the camera to 24p even to make this work with the Elgato HDMI dongle, and Unreal's built in compositing system. This isn't a limitation with SDI based capture like the Blackmagic and Aja cards. Of course, if you just want to get the real time render from Unreal and composite externally in a keyer like a Blackmagic ATEM, this won't affect you. ## Jetset Setup We have Jetset connected as before. We have a Cine calibration loaded. As you can see, we have our Tentacle Sync working and our timecode connected. And currently we have a 24 millimeter calibration loaded up and you can see the yellow reticule that shows the equivalent Cine focal length. It's yellow because we have correctly calibrated. Now we're ready to move forward. We're going to come back to Autoshot, and you can see we have the Jetset device on the network here. We're going to click Open Web Page. And you can see our digital slate is here, timecode is updating. The timecode display here doesn't yet match the Tentacle Sync. That's a bug; we're fixing that. [00:03:00] We're gonna click settings. I'm gonna come down here, and we can pick our external tracking protocol. We can pick either FreeD or LONET2. For Unreal we're going to use LONET2. We're using a 24 frame per second cine camera, so we set this to 24 fPS, and we're going to click set to this PC. We are running Autoshot on the same PC that we're going to be running Unreal on and that's important for this step because we're setting the destination to this IP address So I'm going to go ahead and highlight that click copy. I'm going to save so now we have data streaming out of Jetset. ## Installing LONET Plugins For our next step, we'll install some plugins from LoLed Virtual. I'll put the link below of where they're located. You'll go to this page to their Unreal plugin releases, and download the pre compiled plugins for your version of Unreal, in this case 5. 3. 1. Once you have it downloaded, then what you can do is you're going to right click and extract all And then we're going to open up your Unreal installation [00:04:00] and in this case on my machine it's under D, Epic Games, UE 5. 3 we're going to go into Engine Plugins and then all the way down to virtual production. And we're going to drag the LONET 2 Live Link plugin into virtual production. All right, now we can start up Unreal. ## Enabling Plugins and UI Let's install the needed plugins and go to edit plugins. And we'll type in LONET, make sure our LONET 2 has plugin with the checkbox and type in composure, make sure our composure plugin is checked, type in Python. And we're going to make sure that our Python editor script plugin and our sequencer scripting plugins are both enabled. And if you need to restart, that's a good point to do it. Now what we can do is set up our user interface a bit. We're going to go to our window and virtual production. And make sure that your Composure Compositing, Live Link, and Timecode Provider windows are all checked. ## Unreal Live Link Setup We can go over to our Live Link browser and click plus source pick our LONET [00:05:00] 2 Live Link You. Highlight that, CTRL A to select all, CTRL V to copy. And there is our, there is our real time data coming in. ## Live Render Preview Script We don't have composites yet. So let's go back to Autoshot. And we're going to go to File, Export Unreal Live Preview Script. And this will probably by default say Cam Link, that's fine. You can just click Generate. And down here we can see that it has generated a script. It's already copied into our buffer. I'm gonna go to Unreal, Ctrl V to paste, and this is going to create a composite for us. ## Composure Layers And let's go take a look at the, the elements that it created in Outliner. If you type in the search term Jet, you can see that we've got four new items created. One is a Jetset Live Link Camera. If you click that, it'll show the preview of which way it's pointed. And importantly, you can see that it is pointed the same direction. It has the same framing as our aiming reticule on our Jetset camera. This is a key point. Since the calibration is stored and the offsets are stored in Jetset, what we pick in Jetset will [00:06:00] transmit automatically to Unreal. The next piece we're going to look at is the real time feed from the Cine Camera. This is the CG element of the composite. This is the shot. This is the the components put together, which right now it just shows the live action component cause we haven't started keying it yet. And then again, we have the camera. ## LC Media Player and Texture So let's see where we've created these pieces. We can look in our content browser to see where that was created. And we can see that we have a generated folder under media called Lightcraft. Inside Lightcraft we have an LC media player and an LC media texture. We can double click our media player and ideally this will be working with live video if the correct video device was automatically picked. ## Correcting Video Player UR If it was not, then you can come up here and go to video and click Cam Link 4k. And this will show you the string here. You can hit control a to select all control C to copy. And this would be a string you could paste back into a Autoshot. We'll just show you how to do that right here. And we'll come back here. File export unreal live preview script.[00:07:00] And once again, we'll hit control A and control V to paste it here and we'll generate. And it's recreated the script. So this is if a a different video device is enumerated than the, than the one you want. And then you would just paste this back into the console here. But it, it's working fine. So we don't need to change that. We're going to minimize that. We don't want to close it because then our real time video will stop working. And we'll just minimize that there. ## Live Media Layer Next we can look at our Jetset Cine camera element. And if this is black you'll need to set the media source and its details panel. So down the detail panel we can look at composure input inputs and then our media source. And here it shows LC media text here. That's correct. That's where we want it. And in case it's black, you can set it to LC media texture. ## Keyer Settings Next we're going to set our key here. And we're going to go from our input, we're going to go to our transform and compositing panel. We're [00:08:00] going to go to chroma keying. And we're going to pick our key color. We can tap this here. And pick our key color near to our little bear. Click accept. Okay, so now we have a little keyed bear. We can click this to set a larger screen and then we can adjust this to show our alpha channel. We'll uncheck our red, green, and blue. So you can see here's, here's our bear's little alpha channel. And we can actually adjust our keyer a little bit here. And with our material properties. With our material parameters. And we can say, increase our white clip a little bit to bring him in. And adjust our alpha bias. Slide that a little bit to bring him in. Okay. That's it. We'll call that good for now. And we can trigger our red, green, and blue. And uncheck our alpha. and exit out of our preview window. When we click our Jetset Cine shot, which is our composite, we can see our bear composited [00:09:00] and our live action scene composited into our environment. ## Changing Scene Locators Now a key thing here is what happens when we change scene locators inside our app. So if we click our scene locator, we pick our bridge, for example, we are going to jump to the bridge and you can see our bridge has now adjusted in Unreal to match the view in Jetset. So let's go take a quick look at Jetset. Okay. So you can see our little bear is now on the bridge both in the, in the viewfinder here, and in our Unreal view. So you can rapidly switch around with different scene locators and it'll update automatically in Unreal. You don't need to parent, reparent objects in Unreal as the data is already being transformed in Jetset. ## Changing Cine Lens Calibrations And likewise, we can actually change calibrations. I'm going to click our main menu and I'm going to go to recording. Cine calibration, and this time around I'm going to pick our 70mm calibration. Then we come back into Unreal. We need to pick the Jetset Live Link camera. And you can see that we changed the 70 millimeter lens, but our zoom still says 24. ## Triggering Unreal Lens Update with Checkbox So we need to actually go and uncheck [00:10:00] and check any box in here. So we can even use a constraint aspect ratio and recheck it. Now our zoom says 69. 581, which is our calibrated focal length. And we can click on our cine shot again. And there we go. And now, now our little guy is correctly zoomed in. ## Verifying Focal Length and Sensor Width And we can verify this compared to the Autoshot calibration by toggling to Autoshot, and we can see here that for that lens calibration it was on a 23. 1 sensor width and with a 69. 58mm focal length, was what we measured, and in Unreal, there it is: 23. 1 and 69. 581. Now we're going to switch back to our 24mm. Go here, recording, cine calibration, we'll pick our 24mm. There we go, there's our wider field of view. We'll come back into Unreal, pick our Live Link camera, and once again, it's still stuck at the 70mm focal length, so let's check and uncheck. There we go. Now we have our 24mm focal length. ## Adjusting Timing And now we want to look at doing offsets. So we'll come over, we'll come over to our Live Link and [00:11:00] pick our LONET 2 Live Link. And right now our default timing offset is 0. 0 seconds. If we pan back and forth, that's about as good as we can get with just the Elgato device for now. If you want perfect synchronization at the current time, you need to use an SDI card with matched timecode, and we'll go through that in another tutorial. But this will get us going for now.