ruote log_2010-10-20

2010-10-20 03:02:26 utc hassox jmettraux: ping
2010-10-20 03:02:53 utc hassox hey guys
2010-10-20 03:03:14 utc hassox wondering if there's anyone about I could ask a few questions
2010-10-20 03:37:14 utc jmettraux hassox: hello, I'm back
2010-10-20 05:37:09 utc hassox hey jmettraux
2010-10-20 05:37:13 utc hassox sorry mate
2010-10-20 05:37:15 utc hassox had to step out
2010-10-20 05:37:20 utc jmettraux hello, no worries
2010-10-20 05:37:44 utc hassox I wanted to ask you some qestions about the rails example on your github
2010-10-20 05:37:55 utc hassox http://github.com/jmettraux/theboard/blob/master/config/initializers/ruote.rb
2010-10-20 05:38:00 utc jmettraux ok
2010-10-20 05:38:07 utc hassox I'm not understinading what a 'worker' is when you hand it to the engine
2010-10-20 05:38:12 utc hassox does it do work or just schedule it?
2010-10-20 05:38:18 utc jmettraux both
2010-10-20 05:38:33 utc jmettraux engine and work may schedule work
2010-10-20 05:38:39 utc jmettraux only the worker performs it
2010-10-20 05:39:00 utc jmettraux this board example includes a worker so that it's easy to start
2010-10-20 05:39:25 utc jmettraux depending on the final application and where it's deployed, it may be necessary to run the worker[s] separately
2010-10-20 05:39:49 utc tosch_le hassox: if you don't include the worker (and have no other worker running), you may launch processes, but they will never appear in the processes list
2010-10-20 05:39:59 utc jmettraux exactly
2010-10-20 05:40:09 utc jmettraux the launch order will get stuck in the storage
2010-10-20 05:40:42 utc hassox sorry just talking for 10 minutes
2010-10-20 05:40:44 utc hassox I"ll be right back ;)
2010-10-20 05:40:49 utc jmettraux ok
2010-10-20 05:57:57 utc tosch_le added some info about running a worker to ruote-on-rails' ruote-kit initializer (in the rails 3 branch)
2010-10-20 05:58:36 utc jmettraux tosch_le: hello, great
2010-10-20 05:59:03 utc jmettraux I've added links from theboard to ruote-on-rails
2010-10-20 05:59:10 utc jmettraux still have to push
2010-10-20 06:25:25 utc tosch_le ACTION announces: theboard will be integrated into ruote-on-rails. also, the rails 3 branch will become master and the current master branch will become a new complementary and less well maintained rails 2 branch.
2010-10-20 06:27:42 utc jmettraux :-)
2010-10-20 06:33:35 utc hassox hey jmettraux
2010-10-20 06:33:37 utc hassox sorry about that
2010-10-20 06:33:41 utc jmettraux no worries
2010-10-20 06:34:01 utc hassox ACTION reads back'
2010-10-20 06:34:54 utc hassox so I should use ruote-on-rails
2010-10-20 06:35:13 utc hassox so
2010-10-20 06:35:31 utc hassox if I had wanted a worker process
2010-10-20 06:35:40 utc hassox I would set up the engine _in that process_ like
2010-10-20 06:35:46 utc hassox RuoteKit.engine = Ruote::Engine.new(Ruote::Worker.new(storage))
2010-10-20 06:35:48 utc hassox .?
2010-10-20 06:35:54 utc jmettraux yes
2010-10-20 06:36:32 utc hassox so
2010-10-20 06:36:38 utc hassox sorry man I'm a bit thick on this
2010-10-20 06:36:48 utc jmettraux no problem
2010-10-20 06:36:52 utc hassox when you kick off a process, the non-worker engine just dumps it in the storage
2010-10-20 06:37:03 utc hassox does the worker poll the storage to find out what to do?
2010-10-20 06:37:07 utc jmettraux yes
2010-10-20 06:37:11 utc hassox ah right
2010-10-20 06:37:15 utc hassox that was my missing pice I think
2010-10-20 06:37:25 utc tosch_le some background info: i didn't include a worker by default in ruote-on-rails, as the deployments may be different. it's not guaranteed that the rails server process is running all the time, for example when using passenger.
2010-10-20 06:37:41 utc hassox so, as long as engine and worker share the same storage, they're going to be giving / receiving work
2010-10-20 06:37:42 utc hassox ?
2010-10-20 06:37:46 utc tosch_le yes
2010-10-20 06:37:49 utc jmettraux yes
2010-10-20 06:37:51 utc jmettraux :-)
2010-10-20 06:37:54 utc hassox :)
2010-10-20 06:38:03 utc hassox what about in the case of an amqp distribution
2010-10-20 06:38:09 utc hassox is amqp considered "storage" ?
2010-10-20 06:38:17 utc jmettraux no
2010-10-20 06:38:29 utc hassox is it a participant?
2010-10-20 06:38:34 utc jmettraux more like it
2010-10-20 06:38:51 utc hassox so... how does an amqp participant share the storage?
2010-10-20 06:39:00 utc jmettraux you can bind RuoteAMQP::Participant s
2010-10-20 06:39:19 utc jmettraux in fact there are two parts to it
2010-10-20 06:39:23 utc hassox hrm... do you have a diagram of this anywhere handy?
2010-10-20 06:39:26 utc hassox oh
2010-10-20 06:40:06 utc hassox what's the two parts?
2010-10-20 06:40:25 utc jmettraux ruote-amqp is merely a set of classes for passing workitems over AMQP
2010-10-20 06:40:33 utc jmettraux and back
2010-10-20 06:40:53 utc hassox oh
2010-10-20 06:40:56 utc jmettraux the participant is formatting the workitems and shooting them over AMQP
2010-10-20 06:41:12 utc jmettraux something on the other side fetches the workitems, do something about them, and then reply (or not)
2010-10-20 06:41:17 utc hassox and does it keep track of what it's sent out and not received an answer back?
2010-10-20 06:41:23 utc jmettraux yes
2010-10-20 06:41:28 utc hassox so it can re-try or raise
2010-10-20 06:41:30 utc hassox hmm
2010-10-20 06:41:36 utc jmettraux you can set a timeout
2010-10-20 06:41:40 utc jmettraux or not
2010-10-20 06:41:48 utc jmettraux you can re_apply or cancel the leaf
2010-10-20 06:41:50 utc hassox ACTION is going to attempt to port my app from my hand rolled queue implementation to ruote tomorrow
2010-10-20 06:41:57 utc hassox I see
2010-10-20 06:42:16 utc jmettraux maybe you should start with an exploratory thing or a prototype
2010-10-20 06:42:20 utc hassox tosch_le: does ruote_on_rails have a generator ?
2010-10-20 06:42:33 utc hassox It doesn't say anything about it on the readme
2010-10-20 06:42:41 utc hassox jmettraux: yeah I know :(
2010-10-20 06:42:58 utc hassox unfortunately I'm very time constrained on this atm
2010-10-20 06:43:03 utc tosch_le no, there's no generator, but a freshly written rails template
2010-10-20 06:43:08 utc hassox so I'm going to go for file storage only (no amqp)
2010-10-20 06:43:19 utc hassox and use ruote_on_rails to get me started
2010-10-20 06:43:21 utc hassox ok cools :D
2010-10-20 06:43:26 utc tosch_le (for rails 3)
2010-10-20 06:43:32 utc hassox ja cool
2010-10-20 06:43:56 utc tosch_le there should be some info on that in the readme by now
2010-10-20 06:44:17 utc tosch_le (note: rails templates may be applied on existing rails apps, too)
2010-10-20 06:44:31 utc jmettraux nice
2010-10-20 06:44:38 utc hassox oh... it's a full rails app, not an engine?
2010-10-20 06:45:14 utc tosch_le ruote-on-rails is just an example/starting point for using ruote-kit (and therefore ruote) from within rails
2010-10-20 06:45:21 utc hassox gotcha
2010-10-20 06:45:39 utc hassox jmettraux: I'm really a bit concerned about trying to port it over :(
2010-10-20 06:45:47 utc hassox but it'll be fun!
2010-10-20 06:45:49 utc hassox ;)
2010-10-20 06:45:58 utc hassox tosch_le: where are the workers?
2010-10-20 06:48:36 utc tosch_le there's a rake task to start a worker
2010-10-20 06:48:58 utc tosch_le or you may start one each time you start the rails server process
2010-10-20 06:49:15 utc tosch_le have a look at the ruote-kit.rb initializer
2010-10-20 06:49:33 utc tosch_le i've included some comments on that there
2010-10-20 06:49:37 utc jmettraux hassox: I understand, I'm a bit concerned about someone trying to use ruote in a hurry ;-)
2010-10-20 06:49:46 utc hassox jmettraux: :'(
2010-10-20 06:49:49 utc hassox I'm sorry dude
2010-10-20 06:50:00 utc hassox righto
2010-10-20 06:50:31 utc hassox now that I've rediscovered ruote, the thought of doing all this with my hand rolled solution is not very appealing
2010-10-20 06:50:48 utc jmettraux :-)
2010-10-20 06:50:50 utc hassox and i have 3 other guys coming over this weekend to have a full weekend hackathon on it
2010-10-20 06:51:09 utc hassox I'd hat to have my crapola direct the weekends festivities
2010-10-20 06:51:09 utc jmettraux ouch
2010-10-20 06:51:11 utc hassox hate
2010-10-20 06:51:13 utc hassox ja
2010-10-20 06:51:19 utc hassox so I would like to convert it
2010-10-20 06:51:22 utc hassox there's not much so far
2010-10-20 06:51:31 utc hassox a process when an account signs up
2010-10-20 06:51:37 utc hassox sorry
2010-10-20 06:51:39 utc hassox work flow
2010-10-20 06:51:44 utc hassox and another when a user signs up
2010-10-20 06:51:47 utc hassox that's about it at this stage
2010-10-20 06:51:56 utc hassox but by the end of the weekend there will be a bunch
2010-10-20 06:52:04 utc hassox hence my hurry
2010-10-20 06:52:08 utc jmettraux what kind of domain ?
2010-10-20 06:52:32 utc hassox pm
2010-10-20 06:58:03 utc hassox sure
2010-10-20 06:58:05 utc hassox so
2010-10-20 06:58:13 utc hassox if statements and branching based on some field
2010-10-20 06:58:23 utc jmettraux hassox: if there's noone around (during the week-end), feel free to fire your questions at http://groups.google.com/group/openwferu-users
2010-10-20 06:58:28 utc hassox and I can spec processes yeah?
2010-10-20 06:58:33 utc jmettraux yes
2010-10-20 06:58:34 utc hassox thanx :D
2010-10-20 06:59:14 utc jmettraux the specing is not straightforward to ( == no reference doc )... let me write something quicly
2010-10-20 06:59:16 utc jmettraux quickly
2010-10-20 06:59:37 utc hassox I have a meeting now :(
2010-10-20 07:00:02 utc hassox jmettraux: tosch_le thanx heaps for you help guys
2010-10-20 07:00:10 utc hassox ACTION is going to do a lot of reading tonight ;)
2010-10-20 07:00:13 utc jmettraux you're welcome
2010-10-20 07:00:26 utc hassox will you be about tomorow at some point?
2010-10-20 07:00:42 utc jmettraux yes
2010-10-20 07:00:47 utc jmettraux GMT+9
2010-10-20 07:00:58 utc jmettraux and tosch_le is GMT+1
2010-10-20 07:00:58 utc hassox :O
2010-10-20 07:01:01 utc hassox are you in japan?
2010-10-20 07:01:03 utc jmettraux yes
2010-10-20 07:01:09 utc hassox ACTION is in +11
2010-10-20 07:01:10 utc tosch_le GMT+2 atm
2010-10-20 07:01:18 utc jmettraux tosch_le: summertime ?
2010-10-20 07:01:20 utc tosch_le yes
2010-10-20 07:01:21 utc hassox tosch_le: where about?
2010-10-20 07:01:27 utc tosch_le leipzig, germany
2010-10-20 07:02:11 utc tosch_le i'll be here about 9:00 gmt+2
2010-10-20 07:02:25 utc jmettraux ACTION is out for a coffee
2010-10-20 07:02:59 utc hassox thanx heaps guys
2010-10-20 07:03:00 utc hassox ACTION is out
2010-10-20 07:34:19 utc jmettraux tosch_le: I just pushed my stuff to theboard
2010-10-20 07:34:33 utc jmettraux feel free to merge at will
2010-10-20 07:34:44 utc tosch_le thanks!
2010-10-20 07:34:46 utc jmettraux once done, I'll nuke theboard
2010-10-20 07:34:57 utc tosch_le i'll ping you
2010-10-20 07:35:01 utc jmettraux :-)
2010-10-20 07:35:22 utc jmettraux you want to have a branch for it ?
2010-10-20 07:35:44 utc tosch_le i'll create one
2010-10-20 07:37:19 utc tosch_le i'll be back soon
2010-10-20 08:06:17 utc jmettraux slightly enhanced the process testing doc at http://ruote.rubyforge.org/process_testing.html
2010-10-20 11:13:10 utc hassox anyone about?
2010-10-20 11:13:23 utc tosch_le yes, i'm here
2010-10-20 11:13:28 utc hassox hey tosch_le
2010-10-20 11:13:38 utc hassox ACTION is trying to understand what a worker really is
2010-10-20 11:13:50 utc hassox http://ruote.rubyforge.org/configuration.html <-- workers is just TODO
2010-10-20 11:13:52 utc tosch_le it's a worker ;-)
2010-10-20 11:14:22 utc tosch_le well, a worker has afair no configuration options
2010-10-20 11:14:48 utc hassox so, a worker is the thing that takes from the definition, and passes it to the participants?
2010-10-20 11:14:49 utc tosch_le a worker just sits around and looks regularly into the storage
2010-10-20 11:14:58 utc hassox because the participants are running within the worker?
2010-10-20 11:15:25 utc hassox are participants in the same process as the worker?
2010-10-20 11:15:31 utc tosch_le they get their own thread (or not, it depends on the participant)
2010-10-20 11:16:03 utc tosch_le but the participants are in the same process (note the difference to 'thread')
2010-10-20 11:16:11 utc hassox ja
2010-10-20 11:16:14 utc hassox so
2010-10-20 11:16:30 utc hassox can you fire up a worker with participant x,y,z and another with a,b,c ?
2010-10-20 11:17:51 utc tosch_le no, participants are bound to the storage
2010-10-20 11:18:23 utc hassox so
2010-10-20 11:18:36 utc hassox for a given storage, you have to have all participants running in a single process?
2010-10-20 11:18:43 utc tosch_le but you may bind a participant to a specific engine/worker combination
2010-10-20 11:19:00 utc hassox is there any examples of doing that?
2010-10-20 11:19:06 utc tosch_le you may have more than one worker :-)
2010-10-20 11:20:49 utc tosch_le have a look at http://ruote.rubyforge.org/participants.html, section "participants and threads"
2010-10-20 11:21:22 utc hassox yeah that's the next page I'm on
2010-10-20 11:23:05 utc tosch_le to bind a participant to an engine/worker combination, register an instance of the participant class, not the class itself. doing that is not really recommended, though.
2010-10-20 11:24:36 utc tosch_le if you want to know the worker better, have a look at its code, it's really straightforward: http://github.com/jmettraux/ruote/blob/ruote2.1/lib/ruote/worker.rb
2010-10-20 11:24:46 utc hassox cools thanx
2010-10-20 11:25:51 utc tosch_le the step method is where the work happens
2010-10-20 11:26:07 utc hassox read worker source first, or participant doc
2010-10-20 11:27:00 utc tosch_le participant doc, than source. but it doesn't matter, i suppose.
2010-10-20 11:27:06 utc hassox kk
2010-10-20 11:27:08 utc tosch_le s/than/then
2010-10-20 11:35:02 utc codebeaker hey hassox: great work on Warden… I'm a huge fan
2010-10-20 11:35:08 utc codebeaker
2010-10-20 11:35:11 utc hassox hey codebeaker
2010-10-20 11:35:13 utc hassox thanx dude :)
2010-10-20 11:35:29 utc hassox you use it with rails / sinatra?
2010-10-20 11:35:33 utc codebeaker Padrino and Rails
2010-10-20 11:35:39 utc hassox ah cools
2010-10-20 11:35:55 utc hassox ACTION loves that I was able to integrate omniauth into it ;)
2010-10-20 11:36:02 utc hassox all that goodness for free :)
2010-10-20 11:36:04 utc codebeaker … actually trying out Devise this week… kind of a shame they abstracted the differnet `application` concept… but - hey, it's a rails drop-in, so it has it's own benefits
2010-10-20 11:36:15 utc hassox ja
2010-10-20 11:36:16 utc codebeaker yeah, omniauth is epic… kudos to that firm for open sourcing it
2010-10-20 11:37:06 utc tosch_le hassox: do you have any worries a participant may "slow down" the worker?
2010-10-20 11:37:34 utc hassox tosch_le: in a current app I'm working on, if I were using a participant, it sure would
2010-10-20 11:37:54 utc hassox we're doing 5k AR record inserts with callbacks
2010-10-20 11:38:12 utc tosch_le that's … a lot
2010-10-20 11:38:18 utc hassox ja
2010-10-20 11:38:29 utc hassox the guys were trying to do it in request in a rails app ;)
2010-10-20 11:38:45 utc tosch_le that's … not a very good idea
2010-10-20 11:38:47 utc hassox I split it out to amqp and it's much bettter
2010-10-20 11:39:28 utc tosch_le i'd say: use ruote-amqp, then, but as i never used amqp, i can't say much on that
2010-10-20 11:40:04 utc hassox it's a csv import so I'm distributing each line to a queue, and workers take each one
2010-10-20 11:40:10 utc hassox not sure how I'd do that with ruote-amqp
2010-10-20 11:40:14 utc hassox but anyway
2010-10-20 11:40:22 utc hassox ruote is not likely to make it into that app
2010-10-20 11:41:00 utc codebeaker still hoping I can get to grok ruote well enough to use it in an app I'm working on without cocking my schedule too much
2010-10-20 11:41:09 utc hassox ja me too
2010-10-20 11:41:11 utc hassox so tosch_le
2010-10-20 11:41:19 utc hassox nm
2010-10-20 11:41:22 utc hassox I'll read this stuff first
2010-10-20 11:41:48 utc hassox ACTION is just trying to understand the relationship between engine, worker, participant and remote participant
2010-10-20 11:42:37 utc tosch_le engine: doesn't really do anything, just helps to access the storage in the right way (to launch a process or have a look at the processes list)
2010-10-20 11:43:33 utc tosch_le worker: a) fetches work from the storage and processes it (launch a process, trigger a participant)
2010-10-20 11:44:46 utc codebeaker and a remote participant is "someone" that's offline, out of the loop– a user waiting on worklist or something ?
2010-10-20 11:45:00 utc tosch_le worker: b) watches over schedules so that they are triggered in time
2010-10-20 11:45:20 utc tosch_le where does the term "remote participant" come from?
2010-10-20 11:46:20 utc tosch_le participant: receives a workitem from the worker and does something about that. may eventually return the workitem so that the flow continues
2010-10-20 11:46:22 utc codebeaker not sure, something hassox mentioned - and it's still not clear for me (but I didn't yet thoroughly examine ruote-kit) - how that worklist integrates when one of your particpants is a human
2010-10-20 11:46:28 utc codebeaker ahh "may eventually"
2010-10-20 11:46:59 utc tosch_le the participant doesn't have to return the workitem immediately
2010-10-20 11:47:37 utc codebeaker ahh, that makes sense - so the "worklist" is workitems that are blocked at a certain participant
2010-10-20 11:48:28 utc tosch_le ruote has a storage participant. it receives workitems and puts them into the storage. you may ask the storage participant which workitems it has stored, change them and, if you're ready, tell it to return the workitem to the engine
2010-10-20 11:48:35 utc tosch_le yes
2010-10-20 11:52:25 utc tosch_le "remote participant" as i understand it: a participant registered in the engine receives a workitem. it passes the item to a remote receiver and does not reply to the engine. there is some instance waiting for messages from the remote side, which will do the reply. i already told you i don't know ruote-amqp, but it'll be something like that.
2010-10-20 11:54:05 utc tosch_le kennethkalmer: perhaps you could explain the basics of ruote-amqp, please?
2010-10-20 11:56:07 utc kennethkalmer hi guys
2010-10-20 11:56:12 utc kennethkalmer in a couple of minutes please :)
2010-10-20 11:56:21 utc tosch_le thanks!
2010-10-20 11:57:16 utc tosch_le i said: "the participant doesn't have to return the workitem immediately". it may never do it, too.
2010-10-20 11:59:27 utc hassox tosch_le: sorry just had to do some stuff
2010-10-20 11:59:30 utc hassox yeah that's what I meant
2010-10-20 11:59:46 utc hassox I was looking at daemonkit the other day and there is a ruote-amqp generator in there
2010-10-20 12:00:05 utc hassox tosch_le: so
2010-10-20 12:00:22 utc hassox a participant, could hand the itme over to another engine / worker ?
2010-10-20 12:01:42 utc tosch_le in my explanations: "remote receiver" == outside ruote
2010-10-20 12:02:06 utc tosch_le a participant may pass the workitem anywhere
2010-10-20 12:02:12 utc hassox yeah
2010-10-20 12:02:32 utc tosch_le there's even a participant implementation which does http calls
2010-10-20 12:02:48 utc hassox so a "remote paricipant" would have two components then... a 'participant' and something on the other end that the paricipant hands it to, and then receives a response from
2010-10-20 12:03:08 utc tosch_le yes
2010-10-20 12:03:09 utc hassox does that sound about right?
2010-10-20 12:03:13 utc hassox cool
2010-10-20 12:03:32 utc tosch_le have a look at http://github.com/jmettraux/ruote-amqp
2010-10-20 12:04:08 utc tosch_le there's a participant for sending workitems to a amqp queue and a receiver
2010-10-20 12:08:31 utc hassox so tosch_le
2010-10-20 12:08:33 utc hassox sorry dude
2010-10-20 12:08:42 utc hassox if I have an engine only in my rails process
2010-10-20 12:08:48 utc hassox and I have a seperate worker process
2010-10-20 12:09:01 utc hassox do both places need to load and register the participants?
2010-10-20 12:09:36 utc tosch_le no, just register them within the rails process, the participant list is shared in the storage
2010-10-20 12:10:11 utc hassox so you can just say
2010-10-20 12:10:25 utc hassox engine.register_participant "foo"
2010-10-20 12:10:28 utc hassox in the rails app
2010-10-20 12:10:30 utc hassox ?
2010-10-20 12:10:33 utc tosch_le exception: instanciated participants will have to be registered within the engine the worker is bound to
2010-10-20 12:10:44 utc hassox yeah
2010-10-20 12:11:24 utc tosch_le if engine is an instance of Ruote::Engine, yes; but you'll have to say which participant implementation to use ;-)
2010-10-20 12:11:50 utc hassox even when the engine has no worker bound to it?
2010-10-20 12:11:57 utc tosch_le and regarding the exception: using instanciated participants isn't recommended
2010-10-20 12:15:29 utc tosch_le when registering a participant, you have to say which participant class (or instance) to use
2010-10-20 12:16:03 utc hassox even when you're not in a worker?
2010-10-20 12:16:07 utc tosch_le it doesn't matter where you register a participant, as long as it appear's in the storage's participant list, it's ok
2010-10-20 12:16:19 utc tosch_le yes
2010-10-20 12:16:24 utc hassox kk
2010-10-20 12:17:16 utc tosch_le two cases: engine.register_participant 'foo', Ruote::StorageParticipant <- will work without worker bound to engine
2010-10-20 12:18:16 utc tosch_le participant = Ruote::StorageParticipant.new(Ruote::FsStorage.new); engine.register_participant 'foo', participant <-- won't work and isn't recommended
2010-10-20 12:20:06 utc hassox ah good to know
2010-10-20 12:24:10 utc hassox tosch_le: sorry
2010-10-20 12:24:12 utc hassox still more
2010-10-20 12:24:21 utc hassox with ruote_on_rails
2010-10-20 12:24:25 utc tosch_le no worries
2010-10-20 12:24:27 utc hassox is it safe to have more than 1 worker running
2010-10-20 12:24:39 utc tosch_le it depends on the storage
2010-10-20 12:25:02 utc tosch_le with the default fsstorage and if you're not on windows: it's safe
2010-10-20 12:25:07 utc hassox so
2010-10-20 12:25:22 utc hassox each identical worker won't fight for the same work item?
2010-10-20 12:25:43 utc tosch_le they'll fight, but the first one wins ;-)
2010-10-20 12:25:46 utc hassox or duplicate effort
2010-10-20 12:25:47 utc hassox ok
2010-10-20 12:25:55 utc hassox yeah it's the duplication that I'm concerned about
2010-10-20 12:26:40 utc tosch_le no, there's a blocking mechanism. if a worker grabbed something to do, no other worker will try to do that
2010-10-20 12:27:03 utc hassox awesome :D
2010-10-20 12:27:14 utc tosch_le credits to jmettraux
2010-10-20 12:27:16 utc hassox so if I have a lot of stuff happening, I can just start more workers
2010-10-20 12:27:26 utc tosch_le yes.
2010-10-20 12:27:29 utc hassox saweet
2010-10-20 12:28:00 utc tosch_le you could even start them on different machines (but not with fs storage)
2010-10-20 12:28:20 utc hassox ja
2010-10-20 12:28:24 utc hassox that's what I'm wanting to do
2010-10-20 12:31:04 utc hassox tosch_le: just tell me if I'm anoying ;)
2010-10-20 12:31:15 utc hassox so in a rails app.. could you have 2 engines running?
2010-10-20 12:31:30 utc hassox 1 for in process / request, and another for background async stuff?
2010-10-20 12:31:49 utc tosch_le you're not annoying :-)
2010-10-20 12:32:31 utc tosch_le do you mean 2 engines with different storages or sharing one storage?
2010-10-20 12:32:38 utc hassox different storages
2010-10-20 12:32:58 utc hassox hrm
2010-10-20 12:33:22 utc hassox actually I'm gussing not really... if you had 10 unicorns running rails... they're all going to share the load aren't they
2010-10-20 12:33:25 utc tosch_le i don't see a problem for doing that, but ruote-kit will be at a loss
2010-10-20 12:33:45 utc hassox which means it wouldn't be confined to running _in process_
2010-10-20 12:34:13 utc hassox would that be true?
2010-10-20 12:34:45 utc tosch_le 10 unicorn processes = 10 engines with 10 workers?
2010-10-20 12:35:04 utc hassox that's what I'm guessing
2010-10-20 12:35:22 utc hassox each rails app would start an engine+worker...
2010-10-20 12:35:27 utc tosch_le yes, but they'll share the storage
2010-10-20 12:35:32 utc tosch_le yes
2010-10-20 12:37:24 utc tosch_le that are really advanced questions you're asking (and i'm glad you do it), but unfortunately i'm not 100% up to them – jmettraux is much better in those things. i'd really like you to post (at least some) of your questions on the mailing list
2010-10-20 12:37:55 utc hassox sure
2010-10-20 12:38:00 utc hassox I'll try and remember them :)
2010-10-20 12:38:17 utc hassox ACTION is just reading before bed
2010-10-20 12:38:30 utc tosch_le basically, 10 engines with 10 workers are not really are problem, i suppose they'll be idle often
2010-10-20 12:38:57 utc tosch_le remember: participants are doing there work in an own thread by default
2010-10-20 12:38:59 utc hassox sure but if rails process 1 adds a workflow, then process 2 could actually pick it up
2010-10-20 12:39:09 utc tosch_le yes
2010-10-20 12:39:15 utc hassox k
2010-10-20 12:39:28 utc hassox so it would be unreliable to do it in process in a clustered environment
2010-10-20 12:39:44 utc tosch_le no, why should it?
2010-10-20 12:39:46 utc hassox and expect that _this_ process, because it submitted the workflow, will actually process it
2010-10-20 12:39:58 utc tosch_le yes, that's true.
2010-10-20 12:40:12 utc hassox wait... i'm getting confused ;)
2010-10-20 12:40:37 utc tosch_le ok. waiting ;-)
2010-10-20 12:40:38 utc hassox you're sayint it's true that you can't rely on the process that submits the workflow to actually execute it in a clustered env>
2010-10-20 12:41:08 utc hassox ?
2010-10-20 12:41:22 utc tosch_le the workflow will get executed, but no necessarily by the process which launches it
2010-10-20 12:41:33 utc hassox yup
2010-10-20 12:41:38 utc hassox that's what I wanted to clarify
2010-10-20 12:42:05 utc tosch_le why do you need to pin a workflow instance to a rails process?
2010-10-20 12:43:05 utc hassox if you wanted to wait for the workflow to complete before rendering the page
2010-10-20 12:43:27 utc tosch_le don't do this.
2010-10-20 12:43:30 utc hassox I don't have a specific reason in mind right now... just wanting to see where the boundaries are so I understand how it works
2010-10-20 12:43:39 utc hassox yeah that's the message I'm getting ;)
2010-10-20 12:44:18 utc tosch_le show the user a waiting page and trigger an update when the workflow has completed
2010-10-20 12:44:38 utc hassox ja
2010-10-20 12:44:52 utc tosch_le but it all depends on the workflow and the participants involved
2010-10-20 12:45:05 utc tosch_le in a clustered env, the storage is the bottleneck
2010-10-20 12:45:47 utc hassox ja
2010-10-20 12:45:58 utc tosch_le ruote isn't necessarily designed for speed (there's no need for that when humans are involved – they're just so slow ;-D )
2010-10-20 12:46:32 utc hassox ja
2010-10-20 12:47:07 utc hassox it would still handle a few hundred dispatches per second yeah?
2010-10-20 12:47:25 utc tosch_le btw.: what means 'ja'? i know the german 'ja' == 'yes'…
2010-10-20 12:47:33 utc hassox yeah it means yes
2010-10-20 12:47:37 utc hassox but I'm not geman
2010-10-20 12:47:38 utc tosch_le i didn't do any benchmarks
2010-10-20 12:47:47 utc hassox it's just a habbit I picked up somewhere :|
2010-10-20 12:48:03 utc hassox german even
2010-10-20 12:49:46 utc tosch_le the meego people (http://wiki.meego.com/Release_Infrastructure/BOSS/Design) did some benchmarks, but i can't find them
2010-10-20 12:54:05 utc tosch_le "it would still handle a few hundred dispatches per second yeah?" <- another question for the ml ;-)
2010-10-20 12:55:43 utc hassox :)
2010-10-20 13:15:50 utc hassox tosch_le: thanx heaps for your help tonight :)
2010-10-20 13:25:20 utc tosch_le never mind, it was a pleasure
2010-10-20 14:00:18 utc lbt hassox: http://github.com/jmettraux/ruote/wiki/overview
2010-10-20 14:00:54 utc lbt tosch_le: the benchmarks are at : http://wiki.meego.com/User_talk:Pennymax
2010-10-20 14:01:10 utc tosch_le lbt: thanks a lot!
2010-10-20 14:01:37 utc lbt take them with a large grain of salt... I'm not at all thrilled by the code used to produce them
2010-10-20 14:34:36 utc tosch_le have to leave now. good bye / good night / good evening / whatever to all of you ;-)
2010-10-20 23:43:57 utc jmettraux hassox: I've read through the irc log of yesterday
2010-10-20 23:44:05 utc hassox hey mate
2010-10-20 23:44:06 utc hassox whoa
2010-10-20 23:44:11 utc jmettraux if you have any further question, please fir
2010-10-20 23:44:13 utc hassox there was a lot of stuff in there
2010-10-20 23:44:13 utc jmettraux e
2010-10-20 23:44:17 utc hassox thanx :D
2010-10-20 23:44:19 utc hassox I haven't yet
2010-10-20 23:44:19 utc jmettraux yeah
2010-10-20 23:44:23 utc hassox I had a great read last night
2010-10-20 23:44:29 utc hassox feeling a lot more comfortable this morning
2010-10-20 23:44:35 utc jmettraux feedback on the doc is welcome ;-)
2010-10-20 23:44:45 utc hassox there's lots of TODO sections ;)
2010-10-20 23:44:51 utc hassox praps I can help with those?
2010-10-20 23:45:05 utc hassox I'm still not sure exactly what you can do with a worker
2010-10-20 23:45:28 utc jmettraux I should perhaps replace them with "please ping me on the mailing list if you want this TODO to turn into real doc"
2010-10-20 23:45:47 utc jmettraux as you said yesterday, no worker, no process execution
2010-10-20 23:46:14 utc jmettraux if you have a multiprocess front-end, you might want to not run a worker in there
2010-10-20 23:46:21 utc jmettraux but have 1 or 2 workers in the backend
2010-10-20 23:46:32 utc hassox yeah that's what I thought
2010-10-20 23:47:01 utc hassox so a worker looks at a single storage?
2010-10-20 23:47:05 utc jmettraux yes
2010-10-20 23:47:35 utc hassox and, in the engine (no worker) I have to load the code for participants, even though they're not actually executing?
2010-10-20 23:47:37 utc hassox or
2010-10-20 23:47:53 utc jmettraux not necessarily
2010-10-20 23:47:55 utc hassox if I pass a string to the participant registration, will it not require them to load
2010-10-20 23:48:20 utc jmettraux you can register participants by passing "Hassox::BusyParticipant" instead of Hassox::BusyParticipant
2010-10-20 23:48:24 utc jmettraux exactly
2010-10-20 23:48:32 utc hassox I know these are pretty detailed questions... but I find that it helps me to understand how it all fits together
2010-10-20 23:48:37 utc hassox nice
2010-10-20 23:48:42 utc jmettraux no worries, I learn from your questions
2010-10-20 23:49:00 utc hassox so
2010-10-20 23:49:08 utc hassox lemme look it up
2010-10-20 23:49:29 utc hassox http://ruote.rubyforge.org/configuration.html
2010-10-20 23:49:43 utc hassox in that page, there a table with the various storage backends
2010-10-20 23:49:51 utc hassox with a relative speed rating
2010-10-20 23:50:32 utc hassox If in a process with no worker, how many average work flows could be fired off (ballpark)
2010-10-20 23:50:43 utc hassox 10s, 100s, 1000s
2010-10-20 23:50:51 utc hassox per second
2010-10-20 23:51:31 utc jmettraux a process with no worker,... that means it's just a write to the storage
2010-10-20 23:51:42 utc jmettraux let's bench that
2010-10-20 23:52:16 utc hassox ACTION is thinking redis, since I'm using AR and that kinda puts DM out of the question for the moment
2010-10-20 23:52:42 utc hassox although tbh, I'd prefer to use the db
2010-10-20 23:52:57 utc jmettraux datamapper is fine with AR, IIRC
2010-10-20 23:53:03 utc jmettraux non-intrusive, ne ?
2010-10-20 23:53:03 utc hassox oh cools :D
2010-10-20 23:53:12 utc hassox ah yeah.. it can use AS now can't it
2010-10-20 23:54:23 utc jmettraux not 100% sure
2010-10-20 23:56:28 utc hassox yeah it can
2010-10-20 23:57:15 utc jmettraux bueno
2010-10-20 23:58:34 utc jmettraux http://gist.github.com/637617
2010-10-21 00:00:02 utc jmettraux ruby 1.9.2p0 - FsStorage - 1000 launches - real 8.98 on my gear
2010-10-21 00:00:32 utc hassox 3 seconds on 1.8.7 on my laptop
2010-10-21 00:01:13 utc jmettraux whoah
2010-10-21 00:01:37 utc hassox 2.6
2010-10-21 00:01:38 utc hassox ssd ;)
2010-10-21 00:01:58 utc jmettraux I'm jealous
2010-10-21 00:02:01 utc hassox so about a 100 launches / second on your machine
2010-10-21 00:02:04 utc hassox yeah they're good ;)
2010-10-21 00:02:18 utc jmettraux 2.81 with 1.8.7, oh
2010-10-21 00:02:33 utc jmettraux 1.8.7p249
2010-10-21 00:02:53 utc hassox on your machine?
2010-10-21 00:02:55 utc hassox whoa
2010-10-21 00:03:00 utc jmettraux yes
2010-10-21 00:03:03 utc jmettraux weird
2010-10-21 00:03:07 utc hassox lemme gtry 1.9
2010-10-21 00:03:25 utc hassox need ruote on 1.9
2010-10-21 00:03:56 utc hassox you can always run multiple engines / storage if you need more performance though hey
2010-10-21 00:04:01 utc jmettraux 1.9.1p378 is good though
2010-10-21 00:04:10 utc jmettraux 1.9.2 is a veal
2010-10-21 00:04:11 utc hassox a users engine, an account engine etc
2010-10-21 00:04:30 utc jmettraux but we're only measuring the "launch" speed here ;-)
2010-10-21 00:04:44 utc hassox damnit... I get a segfault on my 1.9
2010-10-21 00:04:49 utc hassox I've done something not fun
2010-10-21 00:05:48 utc hassox could you do that ^ ^
2010-10-21 00:06:02 utc jmettraux the segfault ?
2010-10-21 00:06:07 utc hassox nahy
2010-10-21 00:06:12 utc hassox you can always run multiple engines / storage if you need more performance though hey
2010-10-21 00:06:13 utc hassox ?
2010-10-21 00:06:17 utc jmettraux yes
2010-10-21 00:06:23 utc jmettraux execution performance
2010-10-21 00:06:26 utc hassox whoa 8.6 on 1.9.2 also
2010-10-21 00:06:53 utc hassox execution performance can be split to many workers though.. so you can just add more workers to get more performance yeah?
2010-10-21 00:07:00 utc jmettraux yes
2010-10-21 00:07:10 utc hassox cools
2010-10-21 00:07:35 utc hassox so, another question I had for you
2010-10-21 00:07:39 utc hassox when deving in rails
2010-10-21 00:08:03 utc hassox would you, in your development.rb file, setup the engine with a worker in it so that things could be done immediately
2010-10-21 00:08:15 utc hassox and in your production, have them farmed out?
2010-10-21 00:08:42 utc jmettraux sounds good, worth a try
2010-10-21 00:08:55 utc hassox it will still run everything in a thread right?
2010-10-21 00:08:59 utc hassox so it wouldn't be blocking
2010-10-21 00:09:21 utc jmettraux yes, the worker would thus have his own thread
2010-10-21 00:09:26 utc hassox cools
2010-10-21 00:09:30 utc jmettraux ah
2010-10-21 00:09:34 utc hassox that's awe
2010-10-21 00:09:36 utc jmettraux a note about threads and participant
2010-10-21 00:09:37 utc hassox uh oh
2010-10-21 00:09:39 utc jmettraux s
2010-10-21 00:09:47 utc hassox hit me
2010-10-21 00:10:02 utc jmettraux by default, when dispatching to a participant, a thread is created for the participant.consume(workitem)
2010-10-21 00:10:13 utc jmettraux so that this thing doesn't block the worker
2010-10-21 00:10:21 utc hassox coo
2010-10-21 00:10:27 utc hassox that's ok
2010-10-21 00:10:32 utc hassox it's just the dev environment
2010-10-21 00:10:51 utc jmettraux but if the participant class responds_to to do_not_thread and answers true, the consume will happen in the worker thread (blocking the worker)
2010-10-21 00:10:59 utc jmettraux just a note
2010-10-21 00:11:09 utc hassox that might be good to do in the test environment
2010-10-21 00:11:21 utc hassox oh
2010-10-21 00:11:35 utc hassox but the worker is still running in a thread, so that won't make it non blocking for the test
2010-10-21 00:11:41 utc hassox that's ok... just have to bear it in mind
2010-10-21 00:11:43 utc jmettraux exactly
2010-10-21 00:11:57 utc hassox <3
2010-10-21 00:12:03 utc jmettraux some people have issues when dispatching a lot of workitems
2010-10-21 00:12:04 utc hassox really loving this setup dude
2010-10-21 00:12:14 utc jmettraux feedback is welcome
2010-10-21 00:12:15 utc jmettraux cool
2010-10-21 00:12:16 utc hassox can you limit the thread pool?
2010-10-21 00:12:27 utc jmettraux http://ruote.rubyforge.org/participants.html#threads
2010-10-21 00:12:42 utc jmettraux you could hack your own dispatch pool
2010-10-21 00:13:02 utc jmettraux http://github.com/jmettraux/ruote/blob/ruote2.1/lib/ruote/svc/dispatch_pool.rb
2010-10-21 00:13:12 utc jmettraux or monkey patch the current one
2010-10-21 00:13:15 utc hassox the way I usually think is to run each worker single threaded and just runn a heap of them