ruote tmp/log_2012-11-02.html

2012-11-02 02:57:45 utc jmettraux love those operation gems
2012-11-02 16:50:44 utc mburnett i am trying to understand why ruote seems to have quadratic performance with concurrent_iterator, it doesn't seem to matter whether i use hash or redis backends
2012-11-02 16:50:57 utc mburnett is this avoidable?
2012-11-02 21:22:13 utc jmettraux mburnett: hello, how many workers?
2012-11-02 21:23:59 utc jmettraux quadratic compared to iterator?
2012-11-02 21:25:22 utc mburnett jmettraux: the runtime is quadratic as a function of nodes (I ran it up to about 2000)
2012-11-02 21:25:33 utc mburnett it's linear for a normal iterator (as expected)
2012-11-02 21:27:04 utc jmettraux didn't devote much time to optimize the concurrent-iterator, just made sure it's working as specified
2012-11-02 21:27:48 utc jmettraux I guess it's avoidable by avoiding the concurrent iterator
2012-11-02 21:29:02 utc mburnett interesting, i'll try to test a concurrent job without the concurrent-iterator
2012-11-02 21:38:37 utc kindjal Where does one but the Ruote::Amqp::Receiver?
2012-11-02 21:38:45 utc kindjal s/but/put
2012-11-02 21:39:24 utc jmettraux well it's in the ruote-amqp gem
2012-11-02 21:39:24 utc kindjal I have a dashboard that launches a workflow...
2012-11-02 21:39:35 utc kindjal I have a remote worker that does work and replies...
2012-11-02 21:39:42 utc jmettraux do you mean your class that extends Ruote::Amqp::Receiver?
2012-11-02 21:39:46 utc kindjal yes
2012-11-02 21:39:53 utc kindjal is that "part of the dashboard"?
2012-11-02 21:40:24 utc kindjal dashboard.register :rcv, MyReceiver, :exchange => ['direct', ''], :routing_key => 'ruote_workitems', :forget => true
2012-11-02 21:40:25 utc kindjal ?
2012-11-02 21:40:47 utc jmettraux yes, I was about to send you alink to that piece of code
2012-11-02 21:41:03 utc jmettraux https://github.com/kennethkalmer/ruote-amqp#ruoteamqpreceiver
2012-11-02 21:41:04 utc kindjal so… I start the dashboard, and I don't see the Receiver do anything
2012-11-02 21:41:23 utc jmettraux not even connecting to AMQP?
2012-11-02 21:41:26 utc kindjal yes… I have that, more or less
2012-11-02 21:41:49 utc jmettraux ok, so it's connecting, are there jobs on the queue it's subscribed to?
2012-11-02 21:42:48 utc kindjal yes
2012-11-02 21:42:59 utc kindjal does the receiver go into the process definition?
2012-11-02 21:43:02 utc jmettraux no
2012-11-02 21:43:51 utc jmettraux kindjal: ok, let's look at https://github.com/kennethkalmer/ruote-amqp/blob/master/spec/receiver_spec.rb#L64
2012-11-02 21:44:19 utc kindjal looking
2012-11-02 21:44:33 utc jmettraux it's a transient engine (HashStorage) with one participant "toto"
2012-11-02 21:44:54 utc jmettraux toto is an amqp participant, using exchange "direct" with routing key "alpha"
2012-11-02 21:45:06 utc kindjal yes, I follow
2012-11-02 21:45:13 utc jmettraux queue: https://github.com/kennethkalmer/ruote-amqp/blob/master/spec/receiver_spec.rb#L24
2012-11-02 21:45:29 utc jmettraux queue is used by receiver too: https://github.com/kennethkalmer/ruote-amqp/blob/master/spec/receiver_spec.rb#L68
2012-11-02 21:45:50 utc jmettraux https://github.com/kennethkalmer/ruote-amqp/blob/master/spec/receiver_spec.rb#L75-85
2012-11-02 21:46:07 utc jmettraux the process definition is tiny, all an invokation of participant "toto"
2012-11-02 21:46:16 utc jmettraux which will place the workitem on queue alpha
2012-11-02 21:46:26 utc jmettraux where it gets picked by the receiver
2012-11-02 21:46:59 utc jmettraux and the receiver resumes the flow (it turns the message into a workitem back)
2012-11-02 21:47:06 utc jmettraux and the process instance terminates
2012-11-02 21:47:34 utc kindjal one second...parsing.
2012-11-02 21:47:36 utc jmettraux the receiver doesn't appear in the process definition, it's merely, let's say "infrastructure"
2012-11-02 21:48:45 utc kindjal should I see MyReceiver appear in the noisy log output?
2012-11-02 21:49:16 utc jmettraux not really, you should see "rc" messages coming in
2012-11-02 21:50:08 utc kindjal Is pasting 5 lines of code into this IRC considered "bad form"?
2012-11-02 21:50:26 utc jmettraux well yes, wait
2012-11-02 21:50:42 utc jmettraux please use https://gist.github.com/
2012-11-02 21:52:27 utc kindjal https://gist.github.com/4004514
2012-11-02 21:52:34 utc kindjal I'm expecting to see "msg received" someplace
2012-11-02 21:53:51 utc jmettraux never gets called?
2012-11-02 21:54:35 utc jmettraux ok, quick question, on what system are you testing? Max
2012-11-02 21:54:43 utc jmettraux MacOSX or GNU/Linux?
2012-11-02 21:55:18 utc kindjal This is on Linux
2012-11-02 21:56:45 utc jmettraux does this yield the same [non-]result? https://gist.github.com/4004549
2012-11-02 21:57:04 utc kindjal trying
2012-11-02 21:57:31 utc kindjal yes
2012-11-02 21:57:57 utc kindjal rabbitmq has an "alpha" queue and a "ruote_workitems" queue… I try both with the same result
2012-11-02 21:58:24 utc kindjal alpha is empty, as my remote participant seems to be properly doing workitems...
2012-11-02 21:58:41 utc jmettraux could you show me the whole example via a gist?
2012-11-02 21:58:41 utc kindjal ruote_workitems queue continues to grow as messages go into it and none are "received" by anything
2012-11-02 21:58:48 utc kindjal sure standby
2012-11-02 21:58:49 utc jmettraux ah
2012-11-02 21:59:22 utc jmettraux wait
2012-11-02 21:59:36 utc jmettraux you've set your receiver to :launch_only
2012-11-02 21:59:47 utc jmettraux it doesn't accept workitems, only launchitems
2012-11-02 22:00:07 utc kindjal hmm
2012-11-02 22:00:10 utc kindjal I updated https://gist.github.com/4004514
2012-11-02 22:00:14 utc kindjal let me try that mod
2012-11-02 22:00:43 utc kindjal same result
2012-11-02 22:01:01 utc kindjal sorry for the ugly example… just lots of trial and error.
2012-11-02 22:01:09 utc jmettraux ok, let me build a bench
2012-11-02 22:02:00 utc kindjal fixed a couple paste oddities on that gist.
2012-11-02 22:08:18 utc kindjal changed by receiver invocation to:
2012-11-02 22:08:18 utc kindjal receiver = MyReceiver.new(dashboard, AMQP::Channel.new.queue('alpha'))
2012-11-02 22:08:25 utc kindjal no change in result
2012-11-02 22:12:25 utc jmettraux kindjal: how does that one behaves? https://gist.github.com/7993f2b4ca42749cb5e2
2012-11-02 22:13:38 utc kindjal it says:
2012-11-02 22:13:38 utc kindjal Updating client properties
2012-11-02 22:13:40 utc kindjal and exits
2012-11-02 22:13:56 utc jmettraux now, run it with "NOISY=true" please
2012-11-02 22:14:21 utc kindjal reading output
2012-11-02 22:14:48 utc jmettraux if it exits immediately, it means it's successful
2012-11-02 22:14:57 utc kindjal yes
2012-11-02 22:14:59 utc kindjal exits 0
2012-11-02 22:15:17 utc jmettraux the last noisy line should contain a red underlined te
2012-11-02 22:15:22 utc kindjal yes
2012-11-02 22:15:22 utc jmettraux for "terminated"
2012-11-02 22:15:23 utc kindjal does
2012-11-02 22:15:32 utc jmettraux ok, let's try your change
2012-11-02 22:17:14 utc kindjal taking a stab
2012-11-02 22:17:39 utc kindjal made a MyReceiver subclass, added def handle, puts "msg rev", $stdout flush...
2012-11-02 22:17:46 utc kindjal no output, and it just hangs
2012-11-02 22:19:12 utc jmettraux works for me with https://gist.github.com/4004660
2012-11-02 22:19:40 utc kindjal got it!
2012-11-02 22:21:01 utc kindjal now…what's up with "termination"?
2012-11-02 22:21:08 utc jmettraux ?
2012-11-02 22:21:11 utc kindjal dboard.wait_for('terminated')?
2012-11-02 22:21:16 utc kindjal you commented that out and put in sleep 5?
2012-11-02 22:21:35 utc kindjal if I wait_for()...
2012-11-02 22:21:37 utc kindjal it sits forever?
2012-11-02 22:21:43 utc kindjal or until timeout?
2012-11-02 22:21:45 utc jmettraux termination never happens
2012-11-02 22:21:51 utc jmettraux so let's not wait for it
2012-11-02 22:22:11 utc jmettraux your receiver just receives the amqp message and does nothing more with it
2012-11-02 22:22:22 utc jmettraux no workflow will terminate
2012-11-02 22:22:43 utc kindjal my custom Receiver would have to do something in def handle()?
2012-11-02 22:22:48 utc kindjal in order to properly terminate?
2012-11-02 22:23:07 utc jmettraux if you study the code of the Ruote::Amqp:Receiver you'll see what happens
2012-11-02 22:23:25 utc kindjal ok
2012-11-02 22:23:30 utc kindjal thanks for getting me this far...
2012-11-02 22:23:31 utc jmettraux did you override handle just to make sure the item is coming back?
2012-11-02 22:23:52 utc kindjal I've got a system now that has a dash -> remote worker that qsubs to SGE with DRMAA
2012-11-02 22:23:57 utc kindjal via rabbitmq
2012-11-02 22:24:00 utc kindjal I'm excited!
2012-11-02 22:24:00 utc kindjal :)
2012-11-02 22:24:16 utc jmettraux did you override handle just to make sure the item is coming back? Or is there another reason?
2012-11-02 22:24:30 utc kindjal yeah just to experiment with Receiver
2012-11-02 22:24:35 utc kindjal in any case
2012-11-02 22:24:36 utc kindjal thanks
2012-11-02 22:24:42 utc kindjal I'm done for the day!
2012-11-02 22:24:57 utc jmettraux ok, great, out of the box it's not necessary to extend the Receiver, the default implementation is sufficient
2012-11-02 22:25:02 utc jmettraux I was afraid
2012-11-02 22:31:45 utc jmettraux there is no join coordination cost in the iterator