Office Hours 2024-09-20

===

Joe: [00:00:00] Hey 

Eliot: Joe. 

Joe: Hey Elliot, how are you? 

Eliot: Good, good. Alright, let's see what we have. I have somebody working on the, uh, on the 360 integration. Sorry, I started working through it to figure out how to approach it. I think we've got a way. And yeah, you know, this We're, we're, we're diving even deeper into synthesize than I, than I knew about.

So, uh, the basic concept is we're, I'm looking to modify our existing, our startup, or we have, it's gonna have to go in a couple of phases, I think. And, um, cause the guy in the video just like imported the 360 and hit auto track. Well, I mean, it's great. And it worked. Um, but there's no scale or direction or positional reference.

It's just going to be kind of random where everything ended up. And of course I want to do it right. So I want to have it import with, in the correct coordinate system with our scan, do the little magic tricks with the like project on the mesh, you know, like all this stuff, then we have a referenced initial [00:01:00] 360 solve, and then I think what we're going to be doing is layering on the, the Cine solve on top of that.

As a, um, uh, and it's, uh, sometimes calls it a multi shot and then, you know, so then, then we detect features in that and then we need to link. So there's like, there's some steps, you know, but on the other hand, it, if we get it right, I think it's going to be fairly, you know, procedural and I don't think you're going to need this for, for everything.

And I, you know, I, I think actually might be able to solve this initial shot without it. Um, but I wanted, it was such a good core idea that I, that I think it's gonna be one of these things that, and especially for some of the stuff we have where people are just going nuts, you know, with the cameras, , 

Joe: I just, well, yeah, no, I feel like I gave you like, which I, you know, I'd be happy to shoot another one if it's helpful.

'cause I felt like that was like such a simple, like e Exactly. I didn't at all need a 360 camera to solve. No, no, it's, it's great. 

Eliot: You wanna start, you wanna start with the, the lower hanging fruit. And like, once you get the fruit, then [00:02:00] you work your way up the fruit tree. 

Joe: Thrilled to hear that. It's like that as a core kind of idea, you know what I mean?

Like that, that it's like, that it's resonated and like, this is amazing to hear. That's awesome. 

Eliot: Oh, no, not at all. This is, it was great to see the video and, uh, cause this is. This is the existing week. We, there's, there's two big weaknesses with, um, uh, with tracking through the main camera lenses. One is it's, you can get a narrow field of view and.

You know, there's like not, not much to do there. There's just so little data. And the second is it, it's out of focus at the time. Right. So, you know, 

Joe: and even if something isn't focused, most of the rest, is it so it's like, yeah, 

Eliot: and then the iPhone has as a wider field of view, but it's the wrong frame rate and, and it's.

You know, and he's still susceptible to, to, you know, it's, it's wider. It's not like man, you know, if you're Donna 24, it's not that much wider. Uh, whereas that three 60 is, I mean, it sees everything. So, um, I think it's a great, a great idea. And I, so, [00:03:00] okay. Yeah, here we go.

Ordered one. So, and I, and I got their software development kit and already looking through it and they've got an iOS link. So, you know, to be fair, this is going to take. Just to like set up your 

Joe: thing. Yeah, this is like, 

Eliot: you know, I don't want to, I don't want to hack it. I want to actually systematically go through and get every piece of this.

Correct. So we're, you know, there's, there's weeks, you know, to do go all through this stuff and, and, and dial it in. Um, I mean, it sounds 

Joe: like it's worth every second, like, you know, I'll be patiently waiting with a smile. 

Eliot: Yeah. I mean, this is just, you know, this is reality where you, you start to get into gnarly situations and, and.

You know, exteriors and all these things that are, that. It's very easy for a single end system to start to like fall apart. Um, they do. I mean, it's it's monocular tracking. It's why it's, you know, um, all right, let's see. I'm just like turning it on. All right, please reconnect the device. I'm just, I'm in the middle of like updating the firmware.

Anyway, we'll, we'll get to that a little bit [00:04:00] later. Um, all right. So, and so on the black magic stuff, we've got, I ordered the, it turns out we need a power cable. I ordered that. So that's, that's going in. Uh, Pat's hooking it up on the stage. It's probably going to be, you know, so the, the schedule things is I'm out most of next week.

Um, you know, I'm taking the kids and, you know, Olivia and stuff are going to be in Hawaii. Yay. So I 

Joe: live in a life here. Come on. Yeah. Oh, that's good. That's 

Eliot: good. And, uh, 

Joe: that's great, man. That's awesome. 

Eliot: But I've queued up the things that are queued up and going are so one, the spherical work, you know, to try to get that in place to the, um, you know, I looked deeper into the after effects, how we're going to do this.

And it turns out, ah, got to read the fine print. The new after effects has a new 3d system. It's based on USD, but it does not read USD. That was like two pages in it does read OBJ. And so what we're going to be doing is our scans. We save them both in USD format and an OBJ format. So I, we're going to be looking [00:05:00] at combining the existing camera track, you know, like the, you know, the, the, the, the one for the normal.

Normal, like, you know, after effects with the new 3d import of the mesh. And I want to see if everything, we can get everything to line up. So the mesh is in the correct coordinate system, the cameras and the correct coordinate and the, you know, the background footage. So it all, it all, you know, so hopefully it'll all line up.

So once again, I look at this, I'm like, we got, frankly, we got ourselves a couple of weeks of work of pounding on coordinate systems and after effects to get it to, you know, to be able to, you know, this is not a tomorrow thing. This is like a. 

Joe: And for that update. I appreciate letting, you know, giving me, I mean, this has been so rewarding coming into every single one of these and just talking the stuff through and do it.

So it's like, man, this is, is mind blowing. It feels so good to like be a part of like, just, you know, it just feels awesome. 

Eliot: This is, I'm so glad to have you, to have you on this because I mean, the suggestions are fantastic, right? This is, this is how you get good stuff is you, you have people trying stuff out and they, you know, one of the things that just cropped in, I don't know if you saw [00:06:00] this on the forums, but somebody.

How to get reflections on the, uh, on the image plane and unreal. Um, yeah, so this, that's worth, that's worth looking at. So, but I mean, we can, I'll, I'll worry about that on another point. 

Joe: Yeah. Yeah. That's funny. I wanted to show you something I had made or just that I I've just been playing with, um, and then it's just been kind of helpful for me to, um, Just begin to kind of, so let's see.

So this was, this was just kind of like in, um, for the, just the Tau thing, just been kind of still playing around. 

Eliot: Right. 

Joe: Trying to figure that out, but I, I feel like I've been getting closer and I, I, I'm getting, I'm understanding with the Cine Offset more and more. Uh, why aren't you, what's going on here?

There's.

Eliot: Oh, this looks, this looks great. [00:07:00] 

Joe: Yeah, why won't it let me put on my content drawer? Reflection captured. Huh. Let me For some reason I'm clicking on things but it's not being reactive. Okay, that Do you see me like clicking on that? It's like not going away? It's 

Eliot: not, not doing a thing? 

Joe: Yeah, it's just not If you need to like, you know Won't let me go to the content drawer?

I'm just gonna restart it real quick. 

Eliot: Yeah, reboot and Usually it's easier than fighting. 

Joe: Yeah. 

Eliot: Fighting through a million lines of code. 

Joe: Yeah, I agree. Um, but so, like, this was my, and I started, like, with the CineOffset. Um, so I realized, like, I don't really need to use my iPhone for reference. Like, it's just for reference.

Like, I think that was something that needed to, like, click with me. Which context? On the phone, like looking at the, uh, like the [00:08:00] image through there, when I loaded the model, I do all that, like, that's just reference. Like, cause I was having trouble, like trying to look at both and compose. 

Eliot: Oh, got it. 

Joe: You know what I mean?

And for me, I was, I was just like, okay. And I was able to just look at, I mean, obviously I don't have that reference. But I was able to finally just focus on getting that shot. And then when I brought it in to unreal or anything like that, like it's, it's good. 

Eliot: Right. 

Joe: And it like sits the way that I want, which was like the main thing that I was struggling with, where I was like, wait, which one is giving me.

The information. 

Eliot: Yeah. Yeah. That, that is, that is turning into a pain point that we have, we're over, over time. We have to fix that. Right. It's, it's another one of those. We're like, okay. Okay. You know, there's, there's a few things that are coming in like hard and fast and repetitively. So one of them actually, and this is a good, the, to have you take a look at this, I'm, I'm working out the design for, um, timeline processing, uh, and, um, [00:09:00] it is.

It is, uh, it's an incremental thing, but it will, I think it'll actually solve the, uh, uh, solve what we're, what we're doing, you know, solve the first order of any, but we don't, we don't need to jump into that right now. I can, I can point it out a little bit later. Um, 

Joe: no, yeah, but, but it was, what's important is that looking through the city.

Camera lens is what works. 

Eliot: Yeah. So 

Joe: I was like that. That's the part that I was like, okay, it's not off. It's still, it was sandwiching and there, so that, that made me happy. And I just had to kind of train my brain like, don't, like, just use that for a reference. Line things up and then like shoot on the city camera.

Make 

Eliot: sure where you are, where you think you are in 3D space. Exactly. And then Exactly. And the city cameras. 

Joe: Yeah. 

Eliot: The correct, the correct reference. That's 

Joe: correct. And yeah, 

Eliot: and we. We got to get a laptop and of that, you know, it's just going to, there's some, there's some work. And, but it's totally, it's totally the correct thing to be doing, right?

There's, there's like [00:10:00] not any two ways about that. That's loud and clear. 

Joe: Yeah, precisely. But what, at least what's good though, is that it's working, you know, it's working in a cool way. Um, so, um, So here, this was just like, you know, and this was one of those where I wish I had more room in my living room, but like, you know, this was just sort of like, that was just tracking with my phone.

So that was, um, so it's like here, you can see him in the shot for some, whenever I play it, it doesn't show the image clip. 

Eliot: That, that is something that was in the forums. So one of the people was working through, uh, I, I had that turned off because for the life of me. I ran into problems when I was trying to render and 

Joe: render.

Right. Yeah. 

Eliot: Render. Uh, so this is worth, um, you, we can actually, we can look through and sit there. 

Joe: Yeah. Some, a suggestion from another, a different tutorial that was through something completely different. Um, it was actually when I was trying to find the spline, like a way to figure out like a [00:11:00] spline animation, which I finally did.

Um, because it looks so good. Thanks. Yeah. Cause forever. I was having like, I was banging my head against the wall the last couple of days, because I would, because like, if you can see my, my, my structure tree here, so I had my spline and then here, let me get out of this so I can kind of, uh, so I had my spline and did that.

So I made the spline and then I had the, the actor handle for. You know, just the car, just for everything to like, just attach something to this line. Then I had, this was the scene lock that I used as the reference point here, the car or scene lock for the car in that. And then this was the scene lock for the shot.

So I had to have two different ones. And then this was the camera animation in that. And when I attached it to the handle that I had going along the spline, It's stuck, but still kept all the camera movement that I tracked, which is exactly what I wanted, right? So [00:12:00] now for a while there, it was driving me nuts where I would do that, but I guess you have to do it in the level sequencer that Jet Set makes for me.

Eliot: Yeah, because that's how it knows what the parenting is. 

Joe: Yeah, yeah, exactly, because I kept seeing those lightning bolts, and anytime I tried creating it, I knew it wouldn't work. So that was driving me nuts, but I finally was just like, oh, okay, it has to be in that level sequencer. Maybe it doesn't, and I just don't.

That's gonna be badass. Yeah, it's going to be six. So now one of the things, um, and then one of the other ones that I could still refine kind of the easing on some of the turning points. Um, but, uh, so I was able to finally, and then I was able to get it rendered. Finally. Um, oh, right. So. The why I'm telling you this is because I was having trouble rendering.

So this spline effect, if you see here, there is a thing called final render. Now, when you're rendering it out, like when I, so I'm looking at this, when I would go [00:13:00] to render it out here, it wouldn't show the spline animation. It would just be the still car on the track and it wouldn't, it wouldn't move. And I was like, what the fuck?

So, and apparently there was something similar that the guy had to code in the blueprint. Where he's like, you know, if here, let me pull this up. It'll probably just be the easiest way. Cause I was like, as you're saying this, this, maybe there's something here that will help with that, which is,

sorry, one second. I'm just looking here because I think this, maybe this might be helpful. Where did I, ah, here we go. So. So in the blueprint class here, right. In this construction view right here for final render, he had it where if it's clicked, uh, if, if the final render is there, then it [00:14:00] renders in, then it will actually in the final render.

If that is not checked off, then it, then the spline will not render. 

Eliot: Okay. 

Joe: And I was like, Oh, so there's a toggle when to show up in the preview and then when to, because When I click final render and then I go to preview it it just here i'll show it to you here. It's just um So I go final final render and like look it doesn't it won't move along as flying 

Eliot: not moving 

Joe: And that was driving me nuts.

I was like what the fuck so, but I didn't have that clicked, right? So when I didn't I was like, okay now it's working It was great but then I would go into my movie render queue and it and in every render it was You Like that until I, I remembered, I was like, Oh, right. I remember them talking about building this for this reason.

And then it doesn't work in the preview, but it works in the render. So I don't know if that, if there's anything helpful there at all for you. 

Eliot: But. Interesting. Interesting. [00:15:00] Interesting. I mean, look at the, uh, let's see, this was one of these interesting bits that.

Joe: So, so that was just one thing that I wanted to share. And then the other was I was curious, right? So I've got this and I got it rendered out. Um, but my only thing is that, and maybe this is because I didn't do, I'm curious if like maybe an EXR is like what I need for this. Being able to export it where I can still comp in the subject behind, like, in 3D space here, because when I render the shot, like, it just flattens his footage, so, like, you see how there's that face guard thing there?

It's like, oh, now I have to, like, roto it out. 

Eliot: Right, right, uh, okay, 

Joe: wait a minute that will, so [00:16:00] anything that has, you know, and for a simple thing, like I think I saw it like in the warehouse one, the one that they shot in the barn where it's like, you know, like they had to render out like the, the, the layers 

Eliot: and stuff, 

Joe: yeah, the layers and stuff like that.

So I was just like, okay, is that the only, 

Eliot: uh, there's another way of doing this. Have you used Cryptomat? 

Joe: No, but that was one of the things that I was, I was, as I was investigating, that was what I kept stumbling on was this thing called crypto mat. 

Eliot: Yeah. So you 

Joe: can kind of see here just so that I can, uh, you know, I'll show you, this was like when I rendered it out.

So, Oh, okay. I guess it's not.

All right. Yeah. So it's kind of see that like I did a loose Roto just around him, but it's obviously not perfect. And it's like, I just want him sitting in that cockpit. 

Eliot: Right, right. So this, [00:17:00] okay, so there's, there's an, I think Cryptomat is going to be, so there's a couple of possible ways of doing this. And I think Cryptomat is probably the one you want.

And what it does is when you enable it and when it renders out, it renders the EXR with a couple of extra channels. And those channels have almost like a per object ID. So then you can inside, and this is we're just see you, you buckle up. We're getting into deep visual effects stuff. And every once in a while, after effects gives us a little bit of a thrill where it's, they're like, what?

Um, but I think you can do it in after effects now, they're, they're getting a lot better on that. So now it gives it on an object ID basis. So the one thing you may have to do is, so for example, that if the, if that whole middle of the car is, is, is one, that whole part around the person is one, you know, that purple area is one piece of geo.

This is the part where you might have to do something like, you know, like cut it a little, you know, go find the polygons split into a piece. [00:18:00] Um, and, and to, you know, this is where, you know, It starts to get a little bit interesting. Um, let me think if there's another way of doing that. 

Joe: Because so here actually there is, um, let's see.

So this is the full mesh of the car, right? But then they have all the separate layers here too. 

Eliot: Okay. So yeah, we can look, we can look at that and see the separate. I 

Joe: have to like rebuild it essentially with these pieces, or is there a way to use this like full one that I have here and just kind of like, or.

Eliot: Is when the um, let's look is the car blueprint 

Joe: car is I think it's a static mesh Here let me see do they have any blueprints let's look in the levels demo build data.

Yeah, I guess I don't see any blueprint[00:19:00] 

Eliot: So let's look at Unreal, and then CryptoMass,

CryptoMass, let's see. But this 

Joe: is 

Eliot: going to be cool though, 

Joe: right? Like, I think it's 

Eliot: going to be sweet. Yeah. And 

Joe: I'm like, you know, I'm just trying. So like, and this is the one, like I'm going to put some more other TAL logos and things like that. But essentially this is, you know, like in a sense, what I would like, the goal is to like have like something like this loopable.

Right. Not necessarily like loopable, but just like start from the same playhead so that they sit down, we roll it, we shoot the thing, stop it, and like that's the clip. 

Eliot: Right, right. 

Joe: Um, 

Eliot: yeah. Let's take a look at, let's see if there is. have here.

Joe: And then [00:20:00] another question that I had, I was curious about was like a camera offset. If there's a way to for it to use that jet set info. So like for example, here, if you're looking at my shot, right, I just didn't have enough space. So I wasn't able to go back far enough. You know what I mean? So like, it starts here.

It was like a purely pragmatic thing, but I was like, okay, well, what if I like did that in unreal? So it was just like, okay, what if I had a wider angle also parented to this? Like, so it's like, yeah. 

Eliot: Yes. Yes. You said you, you sort of move, you put the camera on the same transform axis as the main camera.

You just move it back further. Yeah, it totally works. 

Joe: Like that's what I did here. And I'm like, would that work? 

Eliot: Yeah. Yeah. It totally, I know we've done it in blender. I'm sure there's a way to do it in unreal where you're sort of rephotographing the, the, the, the, you know, you basically rephotographing the image plate, uh, you know, that's tracked off the other camera.

So I'm, I'm [00:21:00] sure there's a way to do it. 

Joe: And then, because then my question would be like, since the distance from the camera is different. Like, would it communicate, would I be able to split it to, because, uh, like to my compositor program, and it'd be right, because if I were to take that info from Jet Set and send it to the compositor, it would be referencing the actual shot versus this, you see what I mean?

Yeah. Like, where would it reference, but I don't know, maybe what you're talking about right now with this, um, with the crypto, Matt, maybe, Maybe that's a solve, maybe there's a solve in there somewhere. 

Eliot: You're, you're getting into a, into, I'll be honest, you're getting into tricky, some tricky effects con country that, um, usually what people end up doing is they actually build something like this in a compositor, right?

So they, you know, you have, they have the, the nuke play and then you build another, another camera on top of it. 'cause you're basically rephotographing, [00:22:00] you know, your projected camera, original, um. And there may well be a way to do this. I'm sure there's a way to do it in After Effects. I, you're just out of the range of stuff that I know how to do.

I'm Blender, Blender. I can tell you how to do it. You know, like you just image plate parent, the camera rent, rerender it. And, and, and, uh, you're, you're way past where I know how to do it in 80. So, um, but we may be learning on this as we go. So, 

Joe: no, yeah. I mean, in terms of, I think like attaching the camera and doing that in 80, I think makes sense.

It's more of like, yeah, just how. How does it know the track? You know what I mean? Like doing, uh, and I'm wondering if maybe what we're talking about here with the, uh, Cryptomat, if it might identify where it is in the position there. So, and then maybe if that train, I don't know, but like, yeah, this is interesting.

So you're saying it might be better to try to figure it out in the compositor versus in Unreal. 

Eliot: So it depends if you're going to try to [00:23:00] render everything in Unreal, right? Um, or if you're going to try to composite it later. Um, and, because you, because ultimately what you, you are doing is, you know, okay.

You are, you're basically, you're, you have your original tracked camera. And what you're basically doing is project, doing a camera projection onto an image plate. And then you're re photographing that image plate with a second camera. And there's a way to do it probably on every 3D, you know, compositing package and 3D package ever.

It's just sort of how do they handle their, how do they handle camera projections? That's, that would be what I would, you know, start to look at Unreal. It just has something called image plate. That's what, that's what we use, right? There's an image plate, shows up and you render it. Um, After Effects, I know, I'm sure there's something there.

And if you look up camera projection After Effects, I'm sure there, that's, that, you know, you can start to, start to see it. Um, yeah, this guy, I think probably the first order thing that you're going to want to do is, I think the, is to [00:24:00] figure out Kriptomat. Um, I, there's, there's two possibilities. One is Kriptomat and two is that that turns out to be technically just a giant pain in the butt.

And the second one is you're rendering layers, you know, and, and you put it together. And of course the problem is. You man, you're gonna, where that seam splits, you know, you're, you're gonna have to be careful with, with your seams. You know, there's the front, the foreground and the background and you render it and, and composite it together.

And it's the same thing that you've probably done a zillion times. It's just, it's just fussing more fussing with it. So Cryptomatte is cool when it works. It's so cool because it's not just object IDs, it's ant alias object IDs. So it's made for compositing, right? So you have this nice soft, you know, where, where, you know, one object goes to another object.

It's like a nice, it's not a chunky edge. It's like a nice, soft, like, you know, like you look at it, you go, that's, that's the map that I want. That is, there's, there's just no two ways about it. So, um, So there, there are, and I, I, I [00:25:00] know we set up the, you know, uh, the initial plugins and stuff we did with extra render passes, um, and we can try it.

You know, I'm going to, I'm going to warn you. I've never done it in under real. So I, I did it in blender. A couple 

Joe: of times with a crypto mat, like, yeah, 

Eliot: just render the XRs with a, with a crypto mat pass and, uh, and then pull them into after effects and. 

Joe: See what happens. 

Eliot: Yeah, let's it's you can just try it and uh, let's see what's going on.

Joe: Let's do that right now. Okay. So, uh, okay, great. So then to do that, um, do I just need to like isolate? No, let's 

Eliot: go into, nope, go into a movie render queue and we're gonna throw some switches. Um, 

Joe: oh, right. Yeah, that's in the save. That's right. It's in the deferred rendering. 

Eliot: It's actually, well, we're going to find out.

Let me just go take a look here. Um, so, you know, we are, we're well into the new, okay, so let's see. We're going to go to the level [00:26:00] sequencer and we're going to click on the movie render queue, the little like, that little icon of that. It's the, the little like, um, slate thing. 

Joe: Oh yeah. I got it. Oh, wait, do you not see my, my screen?

Eliot: Oh, Oh yeah. It's not coming up. Uh, maybe you're sharing just the window instead of the screen. Is 

Joe: your, uh,

internet speed going on a crawl or is it just me? 

Eliot: Um, I think mine's all right. Okay, yeah, 

Joe: so

now let's do this.

Oh, yeah, let me close after effects. Maybe it doesn't like also being open.[00:27:00] 

But yeah, they've been, uh, like, I'll send you some pictures of the, uh, are you able to, uh, oh, is this working now? 

Eliot: Uh, yeah, so the screen share should be working. So let's see. 

Joe: Um, and so let's see, we were in movie render queue. 

Eliot: [00:28:00] Yeah. Where's my render queue. Okay. So then actually that, that doesn't, that looks like a default one.

That's not one of the ones that we generated. Um, 

Joe: Oh, uh, let's see. I mean, I was in the sequencer, right? Isn't it? 

Eliot: Yeah. Well, there's sequencer. Where's ours? Okay. Wait, cause that one is not, it's a J. Okay. Where did it? 

Joe: Oh, sorry. Just cause I added, I was like rendering with this. So did you want like the XR?

Eliot: Yeah. Well, so what's a little bit strange is, so if you look at it, it's, um, It's rendering. So Unreal has a default, uh, movie render queue setting where it saves inside the project. And then we have our own where we, we set up EXRs and, and tell it to render in a specific location. So it looks like you might've deleted the original one, but I mean, you know, we can try it, try it with this.

Um, 

Joe: Oh, I, I didn't even know I did that. Sorry. Yeah. 

Eliot: Uh, so, okay, EXR sequence, so then let's, um, okay, let's go see, [00:29:00] um, do they have

EXR sequence? So where, where is Cryptomat going to be? 

Joe: I thought it was like something with the stencil layer clips, like you add the actor layers. I've been, I've been, uh, I've been, uh, that, that term had come up and I was like, oh, okay. Maybe that's 'cause I was looking at like Z depth passes and various things like that, and I was like, oh, maybe that's how I can get it.

Eliot: Yeah. Let's see. Let's see how we're doing this. Uh,

our, there's 5.4, so let's see. Okay, moving on. I'm on 5. 

Joe: 3 also. 

Eliot: Okay, that's, that should be [00:30:00] similar. Um,

Joe: would it be easier for me to just like, how would I reload your settings? Is that easier for me to do? 

Eliot: You can rerun the, um, uh, the problem of course is if you rerun the, the script, it's going to generate a new one. Uh, it's going to generate a new, a new thing. Um, so let's, before we jump into that, let's, we may end up having to do that, but let's first understand how we're going to do, um, how we're going to do this.

Okay, so we want

I think we're going to hit the plus setting and we're going to do,

let's look at object IDs. Okay. And then I need to[00:31:00] [00:32:00] 

hang on. Um,

all right. I'm going to pull this up real quick so that you can see what I mean. Okay. It's like After Effects dead. 

Joe: Yeah. I think it just doesn't like with, with Unreal when they're both running and I think it just logs up the machine. Unreal's a hog. Doesn't like to share.

Uh, oh, sorry. Did you, was there something you, uh, did you need me to do something or share something? 

Eliot: You know, um, let's see. I'm just, I'm looking at this because I haven't, um, I'll show you what I'm looking at. Uh, because I frankly don't know this. I'm just looking at one of the online things on it, looking for it.

Yeah. 

Joe: [00:33:00] Oh yeah. I was watching his video yesterday actually. 

Eliot: All right. So did, uh, okay. Let's move you under cue and let's see what else we have to do here.

And let me get a better resolution here. All right. So how's he doing that? 

Joe: He said to turn the accumulator alpha on.

So he's, right, he's doing it with PNGs. But that's just for transparency, right? Did that, like, hold volume or shape?

Eliot: We don't want a PNG.[00:34:00] 

Okay, there we go. This is what we're looking for. Can you,

Joe: can you hear this? I can't hear it, but I'm, I'm looking, I'm watching it. 

Eliot: Okay, let me see if I can, I can, uh, let me fix this share. Let me share it, because this, I think this is kind of what we're looking for. All right. Yeah, sound. There we go. Oops.

Two. There we go. Uh, ship sound. There we go. That's the one. So let's see how he's, how we're hooking this up. 

Youtube: And go back to your sequencer, hit the movie render queue. And now we're going to edit this guy. So we can't use JPEG for this. It has to [00:35:00] be EXR because it is a multi layered image with data embedded in it.

So just go to setting EXR and multi layer file. And then in the deferred. No, sorry. And in this plus setting, we will have object IDs, uh, as an option, which is from the plugin that we enabled. So just select that and then you can leave it on full or material or actor. I find full to be the best, but, and material is hit or miss for me.

You can give it a go and test it out, but. I find fold to be the best method, and then we will save as, this guy, no, back to layers, I'll grab this guy, change jpeg to exr, and then I'll just do cryptomap, and then save. Hit accept and render this guy out. Sorry, before you click render, I just realized we edited our actor layers.

We want to make sure we get rid [00:36:00] of our actor layers in our deferred rendering. So just delete these guys and now just save over the preset we made. Now you can hit render. So once that's finished, it's. 

Eliot: Okay.

Oh, they do not make this easy. All right. So let's share your screen. Let's see if we can't get, get at least partway through this. I'm going to call it a success. If we render out EXRs and, and we get, it can open this up and like an EXR viewer and see, see layers. This is going to be a major success. Oh, 

Joe: right.

Yeah. I mean, this would be mega if I'm just like, Whoa, I could just stick people in things and then, Oh, it's, 

Eliot: it's cool. There's a reason why everybody. So, okay. So for, okay, let's see, let me go back to this. I'm looking through his, his series of steps and I'm like, I'll just walk through it while I can see it.

So, okay. So, okay. We have any XR sequence. That's fine. [00:37:00] And instead of, uh, let's see, can you highlight the EXR sequence for a second? All right. So P I Z that's fine. Um, and then

let's click on the game override.

Oh, actually that's a, I'm sorry. Let's click on the XR sequence again. And okay. Multi layer file.

Okay. Object IDs. There we go.

And click on the object IDs. Full is fine. Okay. Go ahead and include the translucent objects. Is that windshield is going to be a thing.[00:38:00] 

All right. 

Joe: There's actually no windshield, but, 

Eliot: oh, so it's not transparent. Okay. So then we don't, we don't need the transparent objects. Nevermind. Cause I let him some complexity. 

Joe: And that's good to know. We can do that though.

Eliot: Let's go back up to the EXR sequence. Um, and highlight that and let's, let's look at what we're calling it lagging a bit. Uh, Oh, Oh, okay. Okay. So, sorry. And then let's click on output because I want to see what file name is generating. 

Joe: Oh, output. Sure. 

Eliot: Yeah. Okay. So [00:39:00] say let's, uh, and the output directory. Okay.

Sequence name. Alright, we know where it's gonna go. Okay, that's fine for now while we're figuring this stuff out. Um, alright, sequence name, frame number. Okay, so that's fine. Let's see what else we have to do here.

Okay. And we can actually, uh, we can just click a preset here, uh, on the upper right hand corner, we can, uh, just click that and just save the preset. Uh, yeah. So this is good as it is. Okay. Yeah. I think that's, that's fine. Let's call it crypto Matt render. Yeah.[00:40:00] 

That sounds good. Let's go accept.

Okay, so then let's go ahead and click render local. Let's see what happens here

Joe: Oh, yeah, I didn't put the I didn't turn the final render on so it's not going to move but that doesn't matter 

Eliot: That's fine. That's fine. Well, uh, we just want to see if it's doing the layers 

Joe: But man, the fact that I didn't have to fucking rig any like lighting reflection The fact that I was like city car, let's do you know what I mean?

Thinking of like that because i'm used to structural hierarchy stuff [00:41:00] Oh, okay. That was fast. 

Eliot: It seems a little fast. Okay. But let's, all right. So let's, um, 

Joe: Cause it's only like 160 frames or something. It's not, it's not a very big shot. It's not a long shot. 

Eliot: All right, so let's go to where those frames were, um, you can go, you can actually open up the movie, right?

That, that, um, yeah, there we go, that, that, that directory, and I think it'll open up, uh, projectors, save, movie renders, or we can open up in Windows Explorer, that's fine too. I think it's right here, right? All right, so what, what program do you have to look at EXRs? Like, if we double click the EXR, what's gonna, what's gonna show up?

Joe: No, let's find out.

Eliot: Photoshop, we might see it.

It's just Photoshop 26, Cdxrs.

Okay.

Alright. I'm not sure if Photoshop can [00:42:00] read Kriptomat by default. Oh, you need a plugin. Um, it's called the Exrio plugin. 

Joe: Exrio. 

Eliot: And look at how recent this is. They may have changed this. 

Joe: 2022, it says there.

Should I try this EXRIO? 

Eliot: Yeah, that seems to be the go to thing of, of, uh, of how people are doing it as of, you know, 2023. So we're pretty current.

Joe: Okay. 

Eliot: Install that. Install. [00:43:00] See if we're seeing layers. 

Joe: Uh, should I open back, should I, uh, I installed it. Should I open back up Photoshop? 

Eliot: Yeah, let's, uh, once you've got the plugin installed, let's, uh, open that up and see if it's, do we need to enable? 

Joe: Let me make sure, maybe I'll, let me try dealing with the official non beta release.

Eliot: Sure. 

Joe: Oh, okay. There we go. Yeah. 

Eliot: Okay. Crypto. Hey, there we go. Cryptomatte masks. That's what we really want. Colorized ID layers. All right. Let's just click open. Let's just kind of see where we're at. And, uh, that's promising. It recognized what it was. 

Joe: Oh. 

Eliot: Yeah. Let's see. Yeah, exactly. No, Cryptomatte.

Cryptomatte's amazing. It's one of those things where you just go, Oh, this is why everybody L everywhere. Okay. So this is cool. So now what we can do is all those different objects are proxy masks. So we can unclick the RGBA and let's, let's walk through the, the, the image of this, just this image. That's just, uh, uh, um, hide all your layers, you know, on all the eyeballs on your layers.[00:44:00] 

There we go. And including the, uh, the color. Oh, actually, you know, we can leave the color on, nevermind. And then let's click on, click on one of these things at a time. Uh, and maybe actually put the color on the bottom just so we see the, uh, the layer on top of it. So let's highlight one of those guys at a time.

Hey, there we go. 

Joe: Yeah. Ah, shit. All right. 

Eliot: So now we can walk our way up and down the polygon masks and look for the one that we're looking for. So that's one. And that's, uh, that's another one. That's okay. All right. This, this is, it's, that's, that's the Cryptomatte mask. 

Joe: This is something. Yes. That's, 

Eliot: that's what we're looking for.

So this looks like the car split into these different pieces of geo. Um, and that's, that's it. So then. All right. So that's promising. So now we know we've got masks. Let's go up into, uh, after effects and let's try it. Yeah. Yeah. 

Joe: Okay. Perfect. Oh man. I'm getting excited a little bit. Not going to lie. Oh yeah.[00:45:00] 

Eliot: And is it okay if this is on, uh, on office hours, I can delete like specific mentions of which project and stuff it is, but this will be super valuable for people to see this, just absolutely. Yeah, no, 

Joe: absolutely. Yeah. If you could, yeah. If you wouldn't mind just blanking any reference to that, that specific.

Yeah. Yeah, sounds great. No problem 

Eliot: projects. Is this? Uh, yeah, this is exactly. 

Joe: Yeah. No, if this can help others, I'm all for it. Um, 

Eliot: and what it tells me is we probably want to tell, um, maybe want to enable, I mean, it wasn't that hard to add that to, to our existing, existing workflow. So maybe I'll put that in.

Joe: I just, uh, just add it to this, to, to this project. Just give it a shot. 

Eliot: Yeah. So let's, uh, let's add in the comp and, or the, the, um, [00:46:00] the, the image sequence.

Joe: All right, show, open the sequence. All right, let's go. Oh, did it matter if it was imported as footage? Should I have chosen something? I noticed that there. 

Eliot: Probably as an image sequence, right? 

Joe: No, yeah, it did, but it, like, because sometimes it does the same thing. Here, let me try one more time. Because it does the same thing with, like, a Photoshop file.

Eliot: Oh, 

Joe: as footage or a composition, which means it preserves all of its Uh, file structure. Oh, I got it. I got it. Yeah, 

Eliot: footage. You probably want that. Yeah, 

Joe: composition, retain layer size. I feel like maybe that is what, um, otherwise it's just going to be a flat clip. Yeah, there we go. Import options, pre composed layers, context sheet.

Okay, I think that that sounds right. [00:47:00] Yeah, it's a silly thing after effects. Well, I mean, I guess it's nice that it gives you the option if you just wanted this footage or not. Um, okay, so let's see and it should open. 

Eliot: You'll be looking at how they do it over here in this video. 

Joe: Yeah. Okay. So I think it's this one here.

So here's the assemble and then there's the contact sheet. So let's go in here. Okay. 

Eliot: So it's in the 3D channel. Um, so 

Joe: here we go. 

Eliot: So I think you apply an effect to it. So, uh, under effect 3D and then drag and drop the Kriptomat thing onto it. And then it should Uh, so we have 

Joe: effect 3D, uh, sorry, you were saying, um Yeah, 

Eliot: so the effect, uh, and then Where's 3D?

Kriptomat? Kriptomat, yep, and apply it to that, that Which, which layer am I applying 

Joe: it to 

Eliot: though? It should [00:48:00] be the EXR sequence. 

Joe: Oh, just the sequence itself? Like the sequence itself? Yeah, just the actual imported, 

Eliot: imported footage or whatever it ends up being. 

Joe: Yeah. Okay. So that's that let's see. So 3d channel, Krypton mats.

Eliot: Oh, 

Joe: that's it. 

Eliot: That's a Krypton mat mask. 

Joe: Boom. Okay. 

Eliot: Okay. 

Joe: Good. 

Eliot: Kind of amazed that actually all just worked. I'm not used to things working. First time you try it, like, you know, okay, that's, that's a plus. Um, so let me look at, I'll show you what I'm looking at here real quick, just so you can kind of see the, um, uh, see this guy.

So share sound. So let me pull 

Youtube: this over. Outside software like cinema 4d or blender. So if you're not bringing in 3d renders from outside of after effects, then these won't really do anything for you. But the Cryptomatte effect allows us to work with [00:49:00] Cryptomatte passes from 3D renders. So I have this 3D render of my cute little Lego heart, and I've rendered out a Cryptomatte pass with it.

Which shows up as just a black blank layer when I first bring it in. But as soon as I bring out the crypto matte effect, then I see all these pastelly looking colors showing up in the shape of my render. Now I specifically rendered out a material crypto matte, meaning that every material in my scene has its own matte color.

And what this effect allows me to do is generate a matte based on any number of those materials. To use it, I'm just going to have it selected and then click on any color that I want to be highlighted. And I can even shift click on a second color. So now I have the whole heart selected and let's say that that's all I wanted for this pass.

Well, I'll come back over here and we'll see that the layer crypto material was chosen. It made a selection, which I could click on these and type in specific selections, but that's much more work than I need to do when I can just interactively click on any color. So if I wanted to add another one, I'll just shift click here.

Well, let's [00:50:00] say, Oh, actually I didn't want those bricks. Well, I'll hold down alter option and click again to remove that selection. Okay. So this is what I want to have selected and I want a mat generated from this. So I'll come to my output and change it from colors. To let's say matted colors. And now my layer has transparency providing me an alpha mat.

So I could use this layer, change it to alpha mat. And now I have just my heart visible, or if I undo that and change this from matted colors to mat only, then I'll get a luma mat and I could use. The luma mat track mat instead. I'll undo that one more time. And the only other option is matted RGBA. And this would only be useful if you have an alpha channel included in your crypto mat, but I don't.

So I'm just going to choose matted colors or mat only crypto mats are a really powerful feature of 3d software for compositing and other software like after effects, but that's everything you need to know about it. Hey, thanks for watching. 

Eliot: All right, let's go. Let's go back on that one piece at a time.

Joe: Like does that mean, [00:51:00] I'm curious what that means. 

Eliot: All right, so you want to share yours and then we'll work through what we're going to look for is how we pick out one of those maths. And I think this is going to work. I think what may happen is you have the same. The. Okay, there we go. And so then you can, so I, 

Joe: I, I did the, you know, I, I selected, you know, like, so it's like I could, I guess, you know, uh, I guess I want them both though.

Eliot: There we, yeah, there we go. Like we're, you know, we were gonna do our first test on this, and then, yeah. Then, so how do we, okay, lemme look at this again to see how we apply it as a mask.

Because what you'll probably have is the same image sequence in two separate layers, one of which has the cryptomask applied to it and one of which doesn't. And that way you're, you're, and you, you have your live [00:52:00] action footage sandwiched in between the two of them. 

Joe: Right. 

Eliot: Yeah, I think that's, I think that's what you're looking for.

So can you, like, Duplicate the, uh, uh, 

Joe: duplicate the CG sequence, put the clip in 

Eliot: between them, and let's see if that does kind of what we're thinking it's going to do. 

Joe: Yeah, let me, uh, oh shoot, this was Let me, this will just be easier for me. I forgot. Uh, I don't, I just got to get the exact same frame chunk in here to match.

I don't, uh, I think I have it in another after project. So I thought it was this one, but I think it was another one. So I'm just going to 

Eliot: re enters.

Joe: Okay. So if I look here, I should have my test clip[00:53:00] 

frame

where we're,

Oh, okay. There we go. There we go. So that is my, yeah, there's the shot there. Okay, cool. So we're,

we'll send this to After Effects.

This is just so that I get the exact same, come on,

yeah, it's not liking Unreal After Effects and Premiere Open at the same time.[00:54:00] 

Okay, so now that should send that exact clip to After Effects.

Yeah. Okay. So let me close from here now. So it's not slowing us down.

Eliot: Sorry. Hey, no worries. I was going through this. So I understand how they're doing other applying maths in the, uh, 

Joe: Oh, perfect. Good. Uh, okay. Great. So here's the clip. Let's copy that. And we're going to paste it in between both of these. Uh, Oh, that's right. It's 10 80. So I'm going to have to, this is[00:55:00] 

what's going on. Hello.

All right. Okay. That looks pretty close. All right. So there we go. Um, okay. So now, but I need to get it on. So then am I, so then maybe for this one, 

Eliot: you need to key you that dude. So he's, he's showing transparency, 

Joe: right? Yes. Oh yeah. 

Eliot: Okay. And so for this one, so, okay. Yeah. So this is what, so 

Joe: like, what if, let me, so maybe I need to cut one of them 

Eliot: matted colors and then we set it to, what did he, what did he said in here?

Joe: So let,

Eliot: ah, okay. Okay. And so you can pick the one on top and change the, uh, with the, uh, crypto mat and change the, the track mat from no mat, mat to alpha mat.[00:56:00] 

Oh, yeah, let's 

Joe: see what we're doing here. You know, actually, I did all of this already. Here, let me, I think it would just actually be easier for me to do this. Yeah. Okay. Cause I already have it keyed in. Hey, there we go. 

Eliot: Okay. So now I have 

Joe: it keyed in Matted. So now, now let me just. I need to just re import.

It's just faster for me to do it this way. Yeah, 

Eliot: yeah, yeah. Makes sense. 

Joe: So we'll do that. I'm pretty sure that was the Let me just make sure.

Yeah, okay. So yeah, we'll do that. It's a composition.

Yes.

Okay,

do you [00:57:00] do it? Yeah. Yes. Okay, cool. So, assemble RQBA, then here, then we want to go Cryptomats, then come up to, I mean, that's awesome. That's amazing. Okay. So let's see. Wait, what am I? Did it just shift click? Oh, wait, no. I need to go matted colors, right? Matted colors. Oh, wait. No, you can 

Eliot: start off with the colors and then you select them and, uh, you're going to pick which ones are going to be.

Joe: Oh, right, I have to have that selected. Wait, what's Did I Is there another step in between here? I thought last time Let me look 

Eliot: Let me look back at that. Clicked 

Joe: on that. Do I need to enable something?[00:58:00] 

I thought I just clicked on that.

Eliot: Okay. So, when curve map is selected, can you click on the colors? 

Joe: No. Oh.

Oh, okay. I just had to double click. You have to go into the layer itself. 

Eliot: Oh, okay. Got it. Okay. So you can shift. Yep. There we go. 

Joe: Okay. So that's both of those. Um, 

Eliot: and then you can go over here. 

Joe: Let me actually cut this one first. So we'll do that. And then I'll duplicate it. And then I will get rid of that.

Click that. Okay. So now I should have, 

Eliot: and you can set that to, um, the, the thing that you set it to is, uh, in the [00:59:00] layer. is, um, 

Joe: oh, it's only letting, okay, so when I switch it on one, it switches them on both. That's interesting. 

Eliot: Okay. 

Joe: So it's like I have my top one. Let's say I D selected on the top one. I guess it applies it to both.

Eliot: Interesting. Okay. All right. Let's, let's try and let's just make sure that we have the base. So yeah, let's, yeah, exactly. We'll get the track mat. Then we're going to set it to, uh, let's see, 

Joe: let's let me pull our key. 

Eliot: Okay. So then you, for the track, Matt, you can just change that to alpha Matt. And then that will, uh, uh, that should show up correctly on top of him.

Joe: Are you talking, uh, oh, right, the, yeah, 

Eliot: down, down this guy. 

Joe: Okay, so, right, if I do, yeah, that's what, so, uh, so I already have [01:00:00] it set, um, I already have a, it's already doing that. Um, to, oh, 

Eliot: okay. 

Joe: So let me just pre compose this and just flatten it because this is a way instead of applying it, this is like how, like in terms of compositing with after effects, like the rec, it's like you, how you do it to a, uh, to like, uh, you create it like an alpha layer and then you composite with it versus just applying like the key, like directly to the footage.

Okay, apply it to a masked version, like to a version that you create an alpha mat and then use the mat. So it's just, uh, so I should be able to just pre compose this here. And then, okay. So now I have that mat that's one. And then now I should be able to just set this as that. And let's see. Yeah. Okay.

We're getting somewhere there. That's something I just need the inverse of that.[01:01:00] 

So that is,

I'm just curious what, 

Eliot: so for the, we pop into that because then the. Because we want the, um, I guess it's in what he says he has the track mat as the alpha channel, but maybe that we're, I don't understand. Oh, 

Joe: okay. We're getting. No, yeah, we're getting closer here. Um, so it's showing it through. Oh, 

Eliot: oh, I see.

I see. There we 

Joe: go. I think no, almost. So it's doing it against the whole thing, but I only want it. Oh, okay. So then what if I just, uh, 

Eliot: see because [01:02:00] the whole object, yeah, 

Joe: that's what's going on. Yeah, that's what's going on. 

Eliot: So the cool thing is, I mean, the, if you, it's, it's looks like it's working. So it's, it's, it's the mass tracking, all this kind of stuff.

So if that, if that's all working, then, then the next step is like, you have to split the, you know, you have to split the geo where you want the object ID, but that's, that's not now we're in the realm of like it actually transferred, it rendered, it came over, you know, that's.

Joe: That's pretty cool. Um, so here, let's see if I just, let's just get rid of you.

Oh, right. If I get rid of,

now let's,

oh, right. Okay. So, yeah. Okay. [01:03:00] Now we're getting, yeah, there, now that, see, that's closer. 

Eliot: I think the footage might not be, The Quite exactly. Yeah. What's going off 

Joe: the footage there? That's weird. It's like it's doing the inverse it should be coming from,

huh? What the fuck is that about?

Eliot: That's, well, it's, there's just something kind of little weird, like it's off sync by a few friends or something. 

Joe: Yeah. Maybe I, maybe I didn't. Maybe I must have,

Eliot: but the cool thing is like the mats are kind of, yeah, I'll figure 

Joe: out why the sink isn't right on that, but the mask is working, which is awesome. 

Eliot: Yeah. Yeah. This, [01:04:00] this is, this is cool. 

Joe: Yeah. This is really cool. So, right. So I, ideally Ideally, okay, so it doesn't have it on this one. So I'm like, okay, what do I do with this?

Now? Let's see. So I just use that.

Oh, 

Eliot: you know what? 

Joe: Oh, okay. So there we go. At least that gives me sorry. What are we saying? 

Eliot: Oh, hang on. I actually have a. Okay, 10 30. Okay, that's fine. So let's see. Let me uh, 

Joe: oh, right. That would also make sense. Copied.

I mean, that's so[01:05:00] 

why isn't it working here? Because that should be the same exact. So it's So it's in good position now because this should just match the camera. This is like from the straight camera sync, so it's like I didn't put any motion on the shot. So it's working, but it doesn't want to work here. I don't know what this weird animation is doing.

Like why it's going up like that. I don't know where that's 

Eliot: coming from. Is there uh, something that got inherited weird when it came in or 

Joe: Oh, wait a minute. Yeah, what is this just

yeah, it's coming from No, oh [01:06:00] wait, no,

okay. Okay. Yeah. Yeah. Yeah that no, that's that's right now. Okay. So let's see

Eliot: when he added the mask was there Any other things that happened? 

Joe: No. Yeah. That's because at first I thought it was like, Oh, maybe did I have a camera set on this, but I didn't bring in the information from Jetset where it like created a camera object and did that. I was like, this is just a render out of unreal.

So the footage should just match, which it does on this plate, which it does here. So, but yeah, let's see. Okay. So that's looking okay now. Okay. So now let's try this. Okay.[01:07:00] 

I think that's,

it's still off. Like what the fuck is that? Where 

Eliot: you get seconds. So let's see. Um, let's double check our unreal. 

Joe: Yeah. Unless we just didn't export the whole, okay. Yeah. Let me see how long is this clip. Oh, that's why. 30 

Eliot: frames. 

Joe: 30 frames strikes again. Okay. It was. You summed up the bitch. Or

hey, now I got to make this 20.[01:08:00] 

Okay. Let's see.

It kind of looks a little grifty.

Okay. I think that's better. Better. 

Eliot: Uh, and what does it look like in Unreal? Unreal? 

Joe: Yeah, let's see. This is the unreal, right? So yeah, actually I should be able to

Uh,

so let's see. That should be a match. [01:09:00] Okay 203 202

Okay, it's matching. Okay, so at least that is there um even though well, let's let's see because this was kind of a little bit of like the trouble that I was kind of telling you that I had were sometimes when I'd render it like it Would look perfect and unreal But then when I render out Even though it's a 24 P match.

Sometimes it's like, it's like, Oh, it looks right, but it's not right. Interesting. What's he doing?

Eliot: Let's see. And in unreal, if we, if we go back, um, [01:10:00] let's see what the,

Joe: Oh, there we go. Oh, okay. I think it's a frame off. I think sometimes with the rendering, I remember seeing that they were talking about that like in Blender 2 where it's like, it starts in 0, versus 1. So sometimes I just remember they were like in like if you're comping Blender and Unreal stuff they're like you have to offset it a frame because one starts at 0 and one starts at 1.

So now let me just see, I'm curious how,

yeah, that looks, so it's like, I just made that one frame adjustment on the unreal one. 

Eliot: Okay. 

Joe: And that seems to like help a lot. There we 

Eliot: go. 

Joe: That looks much better. That looks right. So I think it's just one frame off. 

Eliot: Interesting. Okay, 

Joe: so just something to note for any of those other compositors, anybody else watching this?

I just, I remember that being [01:11:00] a problem with Blender, so I'm not, so I guess it's a problem here too. Um, so, okay, so now that we're here, so I guess, so if I were to turn,

Eliot: I wonder if, because right now you originally brought that in as a precomp, um, is it supposed to be a precomp or should it just be like footage? I don't 

Joe: know. The footage is it's because here's the footage and it's like, if I were to show you what this is, so it's like, you see, this is just the mat that I 

Eliot: made, 

Joe: so it is the footage.

It's just, 

Eliot: oh, I mean, sorry. The, um, the rendered stuff. Um, 

Joe: yeah, yeah. So it's like, this is, uh, just because if you turn on the key, like this is the same sort of thing and you make it a final, like you can kind of see the difference of like, That's [01:12:00] why they're like, don't apply it to your actual footage. Like just use it as a mat and then do colors.

Like I see these bills separately. Yeah. Yeah. Just so that it, you know, it's cleaner that way. Um, and you don't have any holes or patches or, or just things like on the footage, so like it stick it, like at least there, the bars in front of his face, which is cool. Okay. 

Eliot: And then 

Joe: step. So I mean, really what it is like, this would be working if I could, if I just have to split the car in half, 

Eliot: right?

Joe: I just need to have the front half of the car be what it registers as the alpha. Not the back half. 

Eliot: Right. Right. Yeah. So that you can have them two separate object IDs. Yeah. So I can just have an 

Joe: object ID that just go right over it. Um, 

Eliot: okay. You want to go try that and, and, uh, and see, 

Joe: yeah, it can be, I'm like, okay, let's see, so how would I split this [01:13:00] up?

Just don't know how I would split it up on like in here. So let's take 

Eliot: a look at modeling unreal. I mean, I know my unreal has some modeling things on it and maybe there's some, some useful thing we can do or whether we have to like pop it into something else. 

Joe: Yeah. Let's see. So here's the model 

Eliot: and what happens when we, okay.

So we've got, look, there's the elements. So then we can, you can, let's see, you know what? I wonder if we can do it by material. So let's try, uh, clicking it. There's the material slots. There's element zero and let's try the isolate. Uh, there's a little isolate checkbox there. Is it okay. So let's, let's start looking through this because we did originally, we did object IDs.

I think we can also do material IDs. So maybe there, we can just do a material for, um, I mean, maybe we'll get lucky on this, um, on the one [01:14:00] right now. 

Joe: Oh, does it say what it is? No, it

doesn't. Wheels, 

Eliot: chassis, engine, details, wheels. So, 

Joe: yeah. Okay. So here, this is what, this is the part that I would need, like split. Like I just need it cut. Right.

So it looks like that's all one.

Halo.

Okay, yeah, there's that. There we 

Eliot: [01:15:00] go.

Joe: So, so 25 and I think, what was it, 19,

so 25 and, okay, 25 and 13.

Eliot: Let's see. Now there's, it'll be [01:16:00] interesting to see,

you're, you're well into the area of Unreal, I don't know. Which is, can you actually change some, pick out different pieces of this stuff and change it into different material, split it up, I guess. I look forward to the Yeah, I'm 

Joe: there too, yeah. 

Eliot: I look forward to the discovery. 

Joe: Okay. I'm going to play around with this some more, um, and kind of see, but the crypto mat stuff, very promising.

Yeah. Very awesome. 

Eliot: I think that's the basic direction that you want to go. 

Joe: That's how, however 

Eliot: you implement it. And I could even see like, you know, something where you, you copy a piece of the car and cut off the polygons of everything you don't need and just put it back on, cause then you, you know, whatever makes sense to do it.

I don't, you don't need to build that much of it. 

Joe: No, yeah. And I mean, you know, even on just, uh, you know, I get like, [01:17:00] on a worst case, the main thing was getting the halo mask, but it's like, you know, this, this little Roto or, you know, just kind of. Even if I had to do that manually, not the end of the world, even though I'd rather not do that, you know, I find a way, which is this cutting in half.

But in the meantime, just because I wanted to get a, I think this should hopefully be good enough for me to get a test

Eliot: um, 

Joe: because what's great is that the phone does this already. Like jet set on the phone. Does it already? 

Eliot: And 

Joe: I think 30 frames isn't a problem, like, like the things that are the issue with Jet Set, I think for this is not.

Eliot: Right, right. 30 frames per second, they're fine. What I 

Joe: want to do on like the day of live thing, um, which would then I guess require, I just want to see if I'm like, inching us closer to the, to getting us where we want to be with this. I would have the background that's on the spline that's moving. That would be its own kind of background [01:18:00] layer with that motion.

Or I guess not the motion, I guess maybe the spline path built in, but we still want the motion that we're getting from the camera. So it's like the, that would be the background. And then the car would be the USD layer. Yeah, I think the car is definitely 

Eliot: the USD layer. And then you, you put a scene locator in the driver's seat.

You know, and so that way you can snap that 

Joe: driver's seat. Um, and then I can pretty much just shoot right on jet sets any, um, now, and then will the background match with the parallax also like that will all come in. 

Eliot: If it, this'll be an interesting part to start working through of how we export the background.

You know, if you render like a 360, it's not gonna have that much parallax, you know, like the motion, but not not that much parallax. We try to do it in 3D. That'll be worth trying out. Um, it'll be an interesting experiment to see how much 3D of the [01:19:00] background we can realistically get in the phone and have it look good.

Um, you know, there's, there's, there's some things to try that are, that we'll, we'll want to, you know, um, see what's just kind of see what the best approach is. There's, there's some, uh, cause you know, the heart, you know, unreal, you just do everything in 3d, right, you know, kind of stuff this, we're going to want to play around a little bit with, uh, how we do it.

I wouldn't be surprised if the right approach is a 3d foreground vehicle where we can control the. You know, CG and, and, and, you know, have the, the background rendered as, as a 360 image and just tell the operator like, you know, don't, don't frame down right. 'cause it's gonna look weird. 'cause you, the, the road's gonna look weird.

So just frame, you know, put it on a slider and stay, stay like this, this level. And, uh, so that you, you frame them up correctly and then it'll look awesome, you know? Is that so there would be 

Joe: no way, because ideally I'd like to do it the way you guys did the, the X-Wing cockpit, you know what I'm saying?

Where like. Yeah, you can just give the camera [01:20:00] to somebody and they're like, you know what I mean? And you're just kind of Like that's the same kind of experience you'd like to do, you know, 

Eliot: that's, that could be doable. I think we'll, to do that, what we probably want to do is, um, okay. So, so this is a good, this is a good thing.

So there's the, the, the, you know, the compositing test you're doing. And then in parallel, the next thing you're probably going to want to do is, uh, to get some of that more control over the 3d. I think we're going to want to pull that over into blender. Um, so the, the next thing I think you're going to want to do is do the, the, uh, unreal, the blender thing and, and get, get the, uh, the, the city scene into blender, um, and, uh, and let's, let's try that.

And then, then we, then we have just an enormous amount more control. We have the geometry and then we can, if we were baking it, we're baking it. There's just a lot more control of what we can, uh, do. Okay. So, so your 

Joe: thing, we, we just use this, get it to blender and then kind of like bake, do all the things that we need to do.

Okay. [01:21:00] That makes sense. 

Eliot: Yeah. But for the system, 

Joe: it's rid of the unreal gorilla in the room. 

Eliot: Yeah. Yeah. Cause then, then we just have a lot, a ton more control over the, the, over the geometry and, uh, and we can, you know, some of the materials that come over, they're going to look a little bit different.

They're getting there, you know, there's going to be some approximations, but Then we're, then we're in control. 

Joe: So, well, then maybe what I might do, I mean, cause I did this, I might just see if there's other models that are for blender. Like if I, if there's an F a formula one and then like another formula one car for blender.

So, I mean, I will play with the unreal to blenders because I want to understand that and see what benefit can be gained from that. But from this, for this to. I'm also just going to see if like, we can't just to avoid the problem of, like you said, with materials coming in, right. And 

Eliot: yeah, so let me take a look at something because in that it might actually be worth, [01:22:00] and let me look at like a turbo 

Joe: squid or something, you know what I mean?

Just kind of bugging by what we need. 

Eliot: Yeah. That may be a far simpler. Um, uh, is there, there is, um, okay, this is, this is also worth looking at. Um, all right, let's see, because there is a procedural blender city generator, um, that might be worth checking out. Let me, uh, that let's see. 

Joe: Oh, yeah, I'm seeing tons of formula one car models and blender.

Yeah, so it's like, you know, uh, yeah, this would be, and they're like real ones too. 

Eliot: Yeah, yeah, they're, they're, they're, they're, they're the real deal. And so you, you may be able to, um, and I'll show you another thing. This is interesting. It'll be worth looking at. I'll put this in the chat. 

Joe: The only thing is that you're just making [01:23:00] sure that, um, I have control over like putting logos on things and changing billboardy things, stuff like that, just to whatever our sponsors are.

Um, Like on there, 

Eliot: right? So, okay. So I just sent a link to a, it's a, it's a procedural building, uh, generator and blender that you can like, just, uh, you take a, take a quick look at that link because the, what we can do is just build only like, uh, literally like a circular strip of, of a city. So let me just, yeah, so let's, you can take a look at this and cause you draw out the roads and it constructs a city around it.

So what I'm thinking is you almost build like just A very, very simple. Yeah, right. And then this is all procedural. It's in Blender. So once you get sort of the city structures you like, it's pretty straightforward for us to, um, then then to to control, you know, because then you're not trying to import a whole city.

You build only exactly this tiny little strip that you what you want. So you can repeat it or you loop it or whatever. [01:24:00] And then, then we, you know, work out the, you know, the different pieces of it. And then, then there's a lot of control. In 

Joe: my unreal, like, it's literally just a, like a U. Yeah. It's like, you know, just enough so you can get a turn or two in there like that.

I mean, cause the whole thing, I'm thinking like, it's like a 10 second click. It's like. Just a 10 to maybe 15, Matt, whatever, you know, we'll find that right length. Right. But just something that's like, here you go, shared on Instagram. You know what I mean? Like you were here. It's like, that's what they want.

You know, they're usually just getting a still image or one of those. Boomerang kind of things when they step into that. So it's like this, I like, I wanna pump that on. 

Eliot: Yeah. Because this, this, now we just have a lot of, a lot more control. Okay. You know that because, and it looks, it looks pretty sweet too, just looking at it.

You know, I haven't, I just, I, I know people have been waiting for this for a while, but this is, uh, uh, this, because this, I mean, I 

Joe: like this. I, it doesn't have the like, look of a Formula One track or like with all of the, the other stuff would be my only. I mean, this is awesome just for like, [01:25:00] like a procedural rows.

Like that's amazing. 

Eliot: Yeah. What we may may end up doing is, is let's see. So you want, let's see. So there's. Okay. We'll want to have 

Joe: like these cars look great. 

Eliot: Yeah. Yeah. Those, those start to be, they'll start to look like a win. Um, but I think we might, we can change. I know in blender, we can adjust things, you know, cause it's a geometry nodes.

And so it's instant, instant thing, like objects and stuff. But we can reach into that and start to put in, you know, different things. Like we need to have like a. You know, the formula, um, yeah, you're right. It doesn't have the, the road. Oh, I see. 

Joe: Yeah. You know, I 

Eliot: see. 

Joe: Just like this. Oh, that's 

Eliot: interesting. So let's take a look at that.

Joe: You know what I mean? I'm like 76 bucks. 

Eliot: Like, you know, that's, that's interesting. And, and so is that a, is there have a blend model and stuff on that? Let's take a look at that 

Joe: or 

Eliot: anything even close. Okay. [01:26:00] 43 million polygons. That's fine. Okay. Three DS max. Okay. So that's FBX. Okay. I see. So the three it's a 3ds max model.

Um, Let's see Let me look at this Okay, and now i'm gonna have to get off into like about three minutes. Yeah, no, no worries I got yeah, we got another one this this is 

Joe: an ongoing conversation. You know, this is one of those like yeah Um, no, but the crypto match you got me stoked on that I'm, so glad that we were able to get that working and that that's functional but that's that's huge Yeah, yeah, this, this will be the avenues of how to workflow here.

This is cool. Um, 

Eliot: and on the, the city stuff, what will be worth looking at is whether we can add in, because what, what it's, what it is, is it's a, it's a, it's a procedural system. It's built in blenders, geometry nodes, and maybe we can add in some of the, you know, the F1 pieces of it, um, and to the procedural [01:27:00] system.

I'm not sure this there's, there's some questions of how. Cool. Yeah, I mean, it's, 

Joe: I mean, you know, I'm down to, to continue figuring that out just from like a creative standpoint, but like for this, oh wow, I'm just looking 

Eliot: at you're right. You already have an F1 Monaco track. There we go. That's what 

Joe: I'm saying.

I'm like, you know, for, I mean, that one's 350, you know, it's like if they wanted it to look like that, I mean, it's in Vegas. But like, you know, even this Baltimore 140 buck, you know, it's like, this is all within the realm of like, 

Eliot: I 

Joe: can get this covered. Um, so we have it. And then it's like, all right, you know what I mean?

So, 

Eliot: okay. So what I would ask is, can you look for one that is, cause the trick of course is when it's in 3ds max, it's always in like V Ray textures and stuff. And those actually, it's hard to convert that, um, into something that we have. Are so if you can find something. I if it'll be a blend file or a, um, or do you want something 

Joe: that that's out of blender, not through ES max?

Eliot: Well, [01:28:00] or if it can, if it can export an FBX with textures, we can, we can pull that. That could work. Um, it's just that the material, I'm sorry, writing 

Joe: this down so I don't forget. Yeah. X with textures. 

Eliot: Yeah. The material conversion is always the bear, um, with all this kind of stuff. And you know, at a certain point, you know, we actually ha.

You already have the scene in Unreal and we can try the Unreal to Blender, in which case, we already wrote that texture conversion pipeline, right, to get to, you know, so this is, it's worth experimenting a few different axes, right, it's worth trying the Unreal to Blender, yeah, you know, and just kind of seeing which one, one of them is going to be, like, pretty easy, and I almost, I almost suggest you start with the Unreal to Blender one, because you already have a city you like, and we rewrote that pipeline, and I know how it works.

Um, so, you know, that might be one to try out and also look through one to see if there's a really sweet one that's, that's a blend file. Um, and you look on Sketchfab as well. I'd look on Sketchfab. 

Joe: Okay, cool. Um. 

Eliot: All right, dude, I gotta go. This is great. Oh yeah, do your thing. 

Joe: Lots of [01:29:00] progress. Yeah, lots of progress.

Uh, Elliot, you're the man, dude. Thank you. 

Eliot: All right, talk to you soon. Bye.