Error: attempting to step

When simming I have seen this error twice now. Any idea what I should be doing to avoid it?!

E_Element1: Attempting to step node ‘E_Element1’ from ticks 323760.000000 to 324000.000000, however previous evaluation stepped until 295200.000000

  • Eddy version 2.2.2
  • Nuke version 11.3v2
  • Operating system Centos 7
  • GPU model 1080ti
  • Graphics driver version 418.56

Hi Frank,

This means that somehow the simulation in Nuke has skipped over some frames. It shouldn’t be possible so there must be a bug. Is there anything unusual in your scene or any actions in particular that seem to trigger it? I can also try to reproduce it if you can share the scene.

Thanks,
Niall.

Thanks Niall,
I am using Christoph’s custom template for coloured smoke and am projecting a plate as the initial volume. Increasing the voxel size seems to have fixed it for now but it sounds like that was a coincidence. I will share the setup if/when it happens again (it will be the same setup as shared in some of the other threads).

Cheers,
frank

And it just happened again after 2 hours of siming :frowniJudgin by the errors in the terminal I guess my GPU ran out of memory?!

Eddy[INFO] - E_Cache2: Writing channel data to file /local2/eddy/LUM/LUM_LUM6_00013_010_ELEM6Orrery_eddy/LUM_LUM6_00013_010_ELEM6Orrery_eddy.001306.vdb
Eddy[INFO] - Element stepping from 0.0 -> 0.0176253318787 step: 0 @ 0.0176253318787
Eddy[INFO] - Inserting 927352 new buckets. Sparse grid now has 1224910 active buckets
EddyNuke[ERROR ] - EVALUATION ERROR: Traceback (most recent call last):
File “/ohufx/pipeline/tools/nuke/plugins/linux/scripts/SparseSmokeColorElement.py”, line 219, in evaluate
rasterize_emitter(element, “R”, R_emitter, dt_scale)
DeviceMemoryException: <eddy.DeviceMemoryException object at 0x7f40569a9440>

  • which then caused:
    Node::evaluate failed for node ‘E_Element1’, plug ‘element’.
    EddyNuke[ERROR ] - EVALUATION ERROR: Attempting to step node ‘E_Element1’ from ticks 313680.000000 to 313920.000000, however previous evaluation stepped until 313440.000000
    EddyNuke[ERROR ] - EVALUATION ERROR: Attempting to step node ‘E_Element1’ from ticks 313920.000000 to 314160.000000, however previous evaluation stepped until 313440.000000
    EddyNuke[ERROR ] - EVALUATION ERROR: Attempting to step node ‘E_Element1’ from ticks 314160.000000 to 314400.000000, however previous evaluation stepped until 313440.000000
    EddyNuke[ERROR ] - EVALUATION ERROR: Attempting to step node ‘E_Element1’ from ticks 314400.000000 to 314640.000000, however previous evaluation stepped until 313440.000000

Here is the script:
eddy_sim_issue.nk (1.1 MB)

Yes you’re right, the first error indicates it has ran out of GPU memory. We’ll try and improve the error messages, so the first out of memory error doesn’t get lost in all the subsequent frame errors.

I’ve had that too. I was starting to think there was a RAM problem with my machine/card/drivers - something…sounds like it’s not so uncommon.
What is the likely fix? Frank increased Voxel size, therefore reducing the total number which I’m imagining can solve memory problems - I’ve got a 1080TI and I can make stuff like that happen all too often.

Yeah, I am lucky that I am getting away with a larger voxel size in context of the task, otherwise I’d have no idea how to get around this short of buying a new GPU (I’m on 1080ti as well)

Happy to look at the scene to see if there’s any other optimizations I can spot frank. 1.2 million buckets is a huge amount of voxels so it may be something else sneakily eating up your memory.

Colliders and emitter bounds can eat up memory sometimes and is often overlooked.

thanks Matt, I have optimized my bounds now by multiplying the E_Image with a E_Box and ticking “first input bounds only”.
Any of the scripts I have been posting will show you the setup. It’s the default one that I have to repeat for many elements at the moment.

1 Like

BTW there is also a E_SetBounds node which can be used to change the bounds. Using the E_Box and “first input bounds only” is fine, but the E_SetBounds is a bit more direct.

1 Like

Ah yes, much nicer, thanks! Will use that from now on.

I am running into this issue again with the attached scene.
It runs out of memory on the second frame when trying to write the caches and I don’t know why.
Could somebody check out out to see what I’m missing here?

Cheers,
frank

eddy_sim_issue4.nk (119.4 KB)

Hi Frank,

I tried opening your scene, but its missing the Image file.
Any chance you can attach one that i could use to repro ?

Cheers,
Christoph

If you just render out a Checkerboard with the global format it should show the problem.

Hi frank,
Your Merge here is creating large bounds, that could be it.
image
Activate “First input bounds only” on E_Merge44, that should help.

Also the subtract is creating some negative values fed into density. As a failsafe I usually use a Multiply with 1 outside 0 inside on the mask whenever doing a mask like this.

MB

aaaah, that ol’ chestnut gets me every time. thanks!

1 Like