best machine to renderer?

Discuss stuff not about Indigo.
User avatar
Jambert
Indigo 100
Posts: 545
Joined: Wed Jul 30, 2008 10:08 pm
Location: France, Lyon

Post by Jambert » Sun Jan 11, 2009 5:20 am

ZomB

Yes but a q6600 is enough to make test with some indigo experience... I have a professionnal activity and to me doing test on a render farm is realy much expensive, I'll never do it... but I understand what you mean, it's much confortable to buy is own render farm, 10000$ sounds expensive to me

:D

Sukrim
Posts: 66
Joined: Wed Jan 09, 2008 6:47 am

Post by Sukrim » Wed Jan 14, 2009 1:18 am

Example:
For each node:
PSU:
http://geizhals.at/eu/a378175.html 300 Watts is far too much already... ~30€
Mainboard:
A good µATX AM2+ Board with 16GB DDR2 RAM support + integrated graphics ~60€
RAM:
4*2GB DDR2-800 @10-15€/GB ~80€ or more
Storage:
http://geizhals.at/eu/a281545.html silent, fast, big enough for Windows/Linux + a few big scenes/renders ~55€
CPU:
Dual/Quad/Triplecore? Depends mostly on your needs I guess - if you take more cores on one CPU it's going to be more expensive, on the other hand you save money for RAM/PSU/Mainboard/Storage. I'd say since it's the most important part of your Renderfarm you invest a bit, let's say ~150€ for a medium clocked Phenom Quadcore or even 200€ for a Phenom II

-->~225€ + CPU (let's say 250€ + CPU so you can have a better PSU or RAM for example) + Cooling stuff + cables --> 500€ max per node.

8 Nodes + Network stuff (Gbit Switch, NAS/Fileserver?...) should be around 10k$ and you have 32 CPUs @ around 2,5GHz or a bit more which would mean you have a ~60 GHz machine with 8GB RAM on each node and a combined power consumption of about 2 Kilowatts or less, silent + fast SSDs and maybe upgradable to some cheap AM2 PhenomII CPUs in 1 year as well if needed.

This is calculated rather conservatively and without OS - so if you use Linux or similar stuff you can get even more powerful hardware, on the other hand it would be maybe even a little less powerful if you have to buy licenses + stuff for each node as well.

How would you other guys on the forum build a renderfarm with 10k$? Different? Intels? Multiprocessor? Come on, show us your dreams! :)

User avatar
dougal2
Developer
Posts: 2532
Joined: Wed Nov 15, 2006 8:17 am
Location: South London

Post by dougal2 » Wed Jan 14, 2009 1:42 am

If I were building simple dumb render slaves I would not give them any HDD storage at all, I'd use that budget to get a better processor. Intel quads for me.

I'd set up a fileserver system to support network boot so that all slaves boot identically - also better for making system updates to the slaves - you only have to change the OS image on the fileserver.
(I actually would like to build a network boot system one day, just for fun).

Seeing also that each slave node only needs power for mainboard, ethernet, CPU and RAM. I might fork out for slightly more powerful PSUs (400W ?) and run two slaves on a single PSU. That (if possible) would probably save a bit of expense and energy.

I would use Linux, obviously... this would also allow easier scripting for renderfarm management etc. I would also set it up such that I would never have to log in directly to any slave node (because otherwise you'd have to log into ALL slaves nodes which could be time consuming) - they just boot and run.


If only I had the time, money and need for such a system, this would be a fun thing to try :)

User avatar
suvakas
3rd Place Winner
Posts: 2613
Joined: Mon Sep 04, 2006 11:08 pm
Location: Estonia
Contact:

Post by suvakas » Wed Jan 14, 2009 1:50 am

dougal2 wrote:I'd set up a fileserver system to support network boot so that all slaves boot identically - also better for making system updates to the slaves - you only have to change the OS image on the fileserver.
(I actually would like to build a network boot system one day, just for fun).
Do you know any link to a resource that explains how to do it ?

User avatar
fused
Developer
Posts: 3587
Joined: Fri Sep 22, 2006 7:19 am
Location: Berlin, Germany
3D Software: Cinema 4D

Post by fused » Wed Jan 14, 2009 1:55 am

suvakas wrote:
dougal2 wrote:I'd set up a fileserver system to support network boot so that all slaves boot identically - also better for making system updates to the slaves - you only have to change the OS image on the fileserver.
(I actually would like to build a network boot system one day, just for fun).
Do you know any link to a resource that explains how to do it ?
interested, too.

User avatar
dougal2
Developer
Posts: 2532
Joined: Wed Nov 15, 2006 8:17 am
Location: South London

Post by dougal2 » Wed Jan 14, 2009 3:30 am

No details - i haven't attempted it yet!

But google for 'PXE boot linux' gives a starting point.

this looks like a good result:
http://www.kegel.com/linux/pxe.html

I know that for example the Debian installer can be booted over network.

Sukrim
Posts: 66
Joined: Wed Jan 09, 2008 6:47 am

Post by Sukrim » Wed Jan 14, 2009 5:16 am

dougal2 wrote:If I were building simple dumb render slaves I would not give them any HDD storage at all, I'd use that budget to get a better processor. Intel quads for me.
Well, you could also use jsut small CompactFlash cards or cheap USB-Sticks to boot the nodes, I was just thinking that maybe several renders starting + reading files over network multiplied by eight would be quite stressful for a NAS/Fileserver. But I see your point here.
dougal2 wrote:I'd set up a fileserver system to support network boot so that all slaves boot identically - also better for making system updates to the slaves - you only have to change the OS image on the fileserver.
(I actually would like to build a network boot system one day, just for fun).
Me too, but since I'd rather boot Windows and also have seperate clients WOL would already do.
dougal2 wrote:Seeing also that each slave node only needs power for mainboard, ethernet, CPU and RAM. I might fork out for slightly more powerful PSUs (400W ?) and run two slaves on a single PSU. That (if possible) would probably save a bit of expense and energy.
I don't know of any PSU that has 2 ATX-Connectors and if there is no easy solution (like adapters for Molex-connectors) I doubt that this could be cheaper than 60€. Interesting approach though!
dougal2 wrote:I would use Linux, obviously... this would also allow easier scripting for renderfarm management etc. I would also set it up such that I would never have to log in directly to any slave node (because otherwise you'd have to log into ALL slaves nodes which could be time consuming) - they just boot and run.
Of course, but this could be done with Windows and custom scripts etc. as well since especially Indigo's network rendering stuff is more "let's use that other computer too!" oriented than really suitable for using it in a render farm.


On network booting http://etherboot.org/wiki/index.php would be a good place to start, also the TechTalk is rather interesting!

User avatar
dougal2
Developer
Posts: 2532
Joined: Wed Nov 15, 2006 8:17 am
Location: South London

Post by dougal2 » Wed Jan 14, 2009 5:33 am

Etherboot (and derivatives) seem to be GPL replacements for network board ROMs - most modern network interfaces are already compatible with PXE specifications.

I just tried these instructions on my machine here (debian lenny):
http://www.debian-administration.org/articles/478

Seems easy to set up the boot server, but I couldn't get a client machine to get a network address at boot time. If I let the machine boot from it's HD into XP, it did get an address from this machine, but by that time it's too late. Might be a quirk with the network ROM in that machine - or it might be why Etherboot is necessary. I don't know enough about it yet to know why the machine did not PXE boot.

I haven't any other spare machines here to try it with either :(

User avatar
dougal2
Developer
Posts: 2532
Joined: Wed Nov 15, 2006 8:17 am
Location: South London

Post by dougal2 » Wed Jan 14, 2009 5:40 am

Sukrim wrote:
I don't know of any PSU that has 2 ATX-Connectors and if there is no easy solution (like adapters for Molex-connectors)
I would get right in there with a pair of wire cutters and soldering iron! :)

EDIT: It's been done and you can buy ATX splitters apparently!
http://www.overclock.net/faqs/112866-ho ... power.html

Sukrim
Posts: 66
Joined: Wed Jan 09, 2008 6:47 am

Post by Sukrim » Wed Jan 14, 2009 7:44 am

Hm, sounds interesting - at current hardware prices a setup like this might even be cheaper than a dual-socket approach! :O

Anyways: Someone here who gets more GHz than approx. 60 (conservatively calculated of course) out of 10k$ than me? :wink:

User avatar
dougal2
Developer
Posts: 2532
Joined: Wed Nov 15, 2006 8:17 am
Location: South London

Post by dougal2 » Wed Jan 14, 2009 9:07 am

My pricing for the q6600 system i specced in that other thread works out at 86.8 USD/GHz (12 machines for 10k$) - and that includes buying cases, monitors, keyboards, mice.

If you omit the cost of monitor (biggest cost), XP licenses, keyboards and cases - that comes down to about 52 USD/GHz (15 machines for 10k$)

sukrim, your figure is 166.66667 USD/Ghz - that seems quite high ?!
Last edited by dougal2 on Wed Jan 14, 2009 9:14 am, edited 1 time in total.

User avatar
dougal2
Developer
Posts: 2532
Joined: Wed Nov 15, 2006 8:17 am
Location: South London

Post by dougal2 » Wed Jan 14, 2009 9:13 am

edit: I can add/multiply but apparently I can't read

If you overclock the q6600's to 3.2GHz you get 39 USD/GHz :D

User avatar
dougal2
Developer
Posts: 2532
Joined: Wed Nov 15, 2006 8:17 am
Location: South London

Post by dougal2 » Wed Jan 14, 2009 10:00 am

I decided to add things up properly, each node works out cheaper that I expected. These specs don't include any kind of racking/casing, so not a very practical renderfarm. However I did budget in a little bit for network switch/cables etc:

EDIT: better spreadsheet
Attachments
Picture 1.png
Picture 1.png (76.77 KiB) Viewed 1796 times

User avatar
Zom-B
1st Place 100
Posts: 4684
Joined: Tue Jul 04, 2006 4:18 pm
Location: ´'`\_(ò_Ó)_/´'`
Contact:

Post by Zom-B » Wed Jan 14, 2009 4:26 pm

Very interesting math dougal...

I would really go the shared PSU way!!! Saves money + energy + space + heat + noise :D

Also I would invest into Phenom II on AM3 Boards! you wouldn't have this much bang for your money atm. But in 2010/2011 you can upgrade your renderfarm to some 6/8/16/whatever core AM3 compatible CPU, while your Intel based system is by definition not upgradeable :x


fused wrote:
suvakas wrote:
dougal2 wrote:I'd set up a fileserver system to support network boot so that all slaves boot identically - also better for making system updates to the slaves - you only have to change the OS image on the fileserver.
(I actually would like to build a network boot system one day, just for fun).
Do you know any link to a resource that explains how to do it ?
interested, too.
link
polygonmanufaktur.de

User avatar
ave
Posts: 25
Joined: Wed Jul 16, 2008 10:00 pm

wow!

Post by ave » Wed Jan 14, 2009 11:42 pm

Guys, I am really impressed by the numbers and quality of the answers... thanks a lot! :D
I will start to consider the various options together with our technician and see what comes out...
BTW I think ZomB got it right about the renderfarm option: I think it would be unpratical for daily work. Also, since the company will deal with light design, we want to be fast in trying different solutions as soon as we design them.

Post Reply
41 posts

Who is online

Users browsing this forum: No registered users and 3 guests