Massey Render Farm

Come here for help & support.
Post Reply
10 posts • Page 1 of 1
User avatar
fractal.design
Posts: 51
Joined: Wed Sep 08, 2010 4:38 pm

Massey Render Farm

Post by fractal.design » Wed Sep 08, 2010 4:54 pm

Hi,

I am helping get the render farm at Massey in Wellington under way and we are having a few issues with the Network Manager.

So far we are running Indigo as "nice" slaves on the MacPro's all the time, co-ordinating the slaves with the masters using a network manager on one of the servers.

It seems a bit flakey...

Can the license manager be Windows based for the Mac slaves?
Would the licenses need to be converted?

Also, is there anymore documentation on the networking side of indigo than that I found here: http://www.indigorenderer.com/forum/vie ... 19&t=10188
and that in the technical manuals?

Specifically, how does the process work? Do the slaves get locked? (If we kill a master, how do the slaves know to be free?)

Maybe one of you guys can come up and give us a hand at some stage?

Cheers

User avatar
fused
Developer
Posts: 3648
Joined: Fri Sep 22, 2006 7:19 am
Location: Berlin, Germany
3D Software: Cinema 4D

Re: Massey Render Farm

Post by fused » Wed Sep 08, 2010 5:34 pm

Hi fractal.design,
fractal.design wrote:Can the license manager be Windows based for the Mac slaves?
Yes, a license manager run on windows can give out licenses to mac slaves (and the other way around).
fractal.design wrote:Would the licenses need to be converted?
No, the licenses are not OS specific.
fractal.design wrote:Also, is there anymore documentation on the networking side of indigo than that I found here: http://www.indigorenderer.com/forum/vie ... 19&t=10188
and that in the technical manuals?
Afaik that is all documentation we have, sorry :)
fractal.design wrote:Specifically, how does the process work? Do the slaves get locked? (If we kill a master, how do the slaves know to be free?)
The slaves should never just lock. When you kill a master, all slaves connections to it will be dropped and they become free.
fractal.design wrote:Maybe one of you guys can come up and give us a hand at some stage?

Cheers
Sure, just let us know what time suits you the best and we will come around :)

Cheers!

User avatar
fractal.design
Posts: 51
Joined: Wed Sep 08, 2010 4:38 pm

Re: Massey Render Farm

Post by fractal.design » Thu Sep 09, 2010 10:18 am

Hey,

Ok great, have linked the Massey admin to this post.

We are going to test 2.4.13 (when the Mac version comes out)

I think maybe the hostnames were not setup correctly, so we will try some more ideas,
and if we run into any trouble will let you guys know.

Cheers for the (speedy) help,

Oli.

AdminHam
Posts: 9
Joined: Wed Sep 08, 2010 5:47 pm

Re: Massey Render Farm

Post by AdminHam » Fri Sep 10, 2010 2:50 pm

OK, so using Indigo v.2.4.13 I have setup a LaunchDaemon that runs this command on startup:
/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole -n s -h test-server:7200

Trying to connect to a test Network Master machine, running the same release of Indigo..
These are the error messages from the network slave:

Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.

And the Server:

10/09/10 12:38:05 PM [0x0-0x1f01f].com.apple.Indigo[248] *** process 248 exceeded 500 log message per second limit - remaining messages this second discarded ***
10/09/10 12:38:06 PM [0x0-0x1f01f].com.apple.Indigo[248] Error while talking with 1xx.1xx.1xx.1xx:55964: Invalid hello, received 539
10/09/10 12:38:06 PM [0x0-0x1f01f].com.apple.Indigo[248] Error while talking with 1xx.1xx.1xx.1xx:55965: Invalid hello, received 539


However, if I set the Indigo Network Options on the slave machine, then launch the Indigo Network Client GUI, or run the following command as admin, or any user (other than root) works:
/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole -n s

Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
Connected to Network Manager at test-server:7200 successfully.

Running that command as root user fires up the slave:

Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
ClientDiscoveryPacketSender thread started.
Listening on port 7100

But obviously isn't reading the settings.xml file that the Indigo Client creates, and doesn't connect to any server.

So I'm interested to know if it's possible to get the slave running as root, to enable it to run while no user is logged in (which would be must useful in a computer lab scenario).

User avatar
fractal.design
Posts: 51
Joined: Wed Sep 08, 2010 4:38 pm

Re: Massey Render Farm

Post by fractal.design » Fri Sep 10, 2010 3:28 pm

Hey,
Under OSX u can add commands to the "rc.local" file on the machine so the process will load on boot as root.

AdminHam
Posts: 9
Joined: Wed Sep 08, 2010 5:47 pm

Re: Massey Render Farm

Post by AdminHam » Fri Sep 10, 2010 3:50 pm

Hey Oli,
Launchd essentially replaces the rc.local method when using the option 'RunAtLoad'.

http://developer.apple.com/macosx/launchd.html

Here's what the command looks like as a Launchd plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>IndigoNetworkSlave</string>
<key>ProgramArguments</key>
<array>
<string>/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole</string>
<string>-n</string>
<string>s</string>
<string>-h</string>
<string>test-server:7200</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>

AdminHam
Posts: 9
Joined: Wed Sep 08, 2010 5:47 pm

Re: Massey Render Farm

Post by AdminHam » Thu Sep 16, 2010 5:40 pm

So very close to having network rendering working seamlessly...
Setting up the settings.xml file at: /Applications/Indigo/Indigo.app/Contents/Resources/ with our Network Manager hostname has made the simple LaunchDaemon (in the last post) work a treat for running slaves!

Now we've run into an issue when rendering images above 1000x700.. no slaves are used (anything below that res, & slaves are used)
The Indigo app on the server is licensed, but do we need a separate license for Network Rendering to work above the 'free mode' resolution?

In the Network Manager, we have the following error:
Floating License Status:
Cannot validate licence: Could not find '</licence_data>'

Any help regarding our setup would be muchly appreciated :)
Thanks,
Hamish

User avatar
OnoSendai
Developer
Posts: 6235
Joined: Sat May 20, 2006 6:16 pm
Location: Wellington, NZ
Contact:

Re: Massey Render Farm

Post by OnoSendai » Thu Sep 16, 2010 7:04 pm

Hi Hamish,
The slave computers need to have a valid Indigo licence to help with renderings above 0.7 megapixels.

Does the normal Indigo GUI read the licence when run on the slave computers? If so, and if the slave process runs as a different user, it may be an issue with the slave processes reading from a different user directory when reading the licence file.

AdminHam
Posts: 9
Joined: Wed Sep 08, 2010 5:47 pm

Re: Massey Render Farm

Post by AdminHam » Thu Sep 16, 2010 9:38 pm

Thanks for the quick response!

The Indigo GUI reads the license fine, but the slave process is running as root.
Ahh the joy of running as root user...
Is there any way to let the root user 'see' the license file?

I know the standard procedure would be to run the slave as a 'normal' user... :roll:

AdminHam
Posts: 9
Joined: Wed Sep 08, 2010 5:47 pm

Re: Massey Render Farm

Post by AdminHam » Tue Jun 11, 2013 3:51 pm

OK, resurrecting an old thread I know, but just picking up where we left off and wondering if the sort of functionality we've created could be worked into Indigo as standard? (much like the Windows version).
The new setup is a LaunchAgent which runs at Login Window:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>nz.ac.massey.IndigoLoginWindow</string>
	<key>LimitLoadToSessionType</key>
	<string>LoginWindow</string>
	<key>KeepAlive</key>
	<true/>
	<key>ProgramArguments</key>
	<array>
		<string>/Library/Indigo/IndigoLoginWindow.sh</string>
	</array>
</dict>
</plist>
This then runs a one liner shell script:

Code: Select all

sudo -u admin /Applications/Indigo\ Renderer/Indigo.app/Contents/MacOS/IndigoConsole -n s >>/tmp/Indigo.log
Which runs IndigoConsole as Admin (Settings.xml file has been deployed to all Admin accounts so IndigoConsole runs with a network floating license, connects to our network manager & does a master search broadcast).

This seems to run OK, however we seem to have some machines connecting multiple times to the master, via different ports... Maybe we should specify a port?

The next part of the equation is running the IndigoConsole when users are logged in, after a period of inactivity. For this we are using AppStartSaver http://www.themcdonalds.net/richard/ind ... StartSaver
This is setup with the following ByHost preferences enforced for each user:

com.ve3nvm.AppStartSaver.plist

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>bringFront</key>
	<true/>
	<key>commandOptions</key>
	<string>-n s</string>
	<key>displayStatus</key>
	<true/>
	<key>displayTime</key>
	<false/>
	<key>docsList</key>
	<array/>
	<key>hideOnLaunch</key>
	<false/>
	<key>pathName</key>
	<string>/Applications/Indigo Renderer/Indigo.app/Contents/MacOS/IndigoConsole</string>
	<key>quitOnDisplayDim</key>
	<false/>
	<key>quitType</key>
	<integer>0</integer>
	<key>restartProgram</key>
	<true/>
	<key>sendAppQuit</key>
	<true/>
	<key>updateSeconds</key>
	<integer>5</integer>
	<key>windowAlpha</key>
	<real>1</real>
</dict>
</plist>
com.apple.screensaver.plist

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>CleanExit</key>
	<string>YES</string>
	<key>PrefsVersion</key>
	<integer>100</integer>
	<key>idleTime</key>
	<integer>300</integer>
	<key>moduleDict</key>
	<dict>
		<key>moduleName</key>
		<string>AppStartSaver</string>
		<key>path</key>
		<string>/Library/Screen Savers/AppStartSaver.saver</string>
		<key>type</key>
		<integer>0</integer>
	</dict>
	<key>showClock</key>
	<true/>
</dict>
</plist>
It would be really neat to finally get this running smoothly, and available for all our students to cut down their render times :D

Post Reply
10 posts • Page 1 of 1

Who is online

Users browsing this forum: No registered users and 2 guests