ruote tmp/log_2011-01-20.html

2011-01-20 12:13:11 utc belucid jmettraux, you there?
2011-01-20 12:49:28 utc jmettraux belucid: yes
2011-01-20 12:49:44 utc belucid hey, how are you this (evening?)
2011-01-20 12:49:53 utc jmettraux doing fine and you ?
2011-01-20 12:50:11 utc belucid doing pretty good, had a question for you if you have a couple moments
2011-01-20 12:50:23 utc jmettraux please
2011-01-20 12:50:25 utc belucid cool
2011-01-20 12:50:56 utc belucid so... I'm having a bit of a conceptual block about the correct "ruote way" to achieve something
2011-01-20 12:51:35 utc belucid seems pretty simple, I'm not sure why I'm having such trouble with it, I have RTFM'd quite a bit so it may point to a problem in the FM :-)
2011-01-20 12:51:41 utc belucid here's the scenario
2011-01-20 12:53:02 utc belucid most steps in my workflow are automated, no problem, they are AMQP participants and worked with daemon-kit ruby processes or python-amqp-ruote client processecs
2011-01-20 12:53:36 utc belucid no issue there, they queue up... get worked LIFO, then end up back in ruote_work_items to move onto the next step
2011-01-20 12:53:53 utc belucid but certain steps are done by people... in particular, Amazon Mechanical Turk people
2011-01-20 12:54:37 utc belucid so, in those cases... there is the same sort of thing... a AMQP participant that queues the work up at Amazon
2011-01-20 12:54:50 utc belucid but then the workflow needs to block, until the work gets performed
2011-01-20 12:54:56 utc belucid and that's where I'm getting stuck
2011-01-20 12:55:00 utc belucid following so far?
2011-01-20 12:55:04 utc jmettraux yes
2011-01-20 12:55:06 utc belucid k
2011-01-20 12:55:17 utc jmettraux ruote can wait
2011-01-20 12:55:25 utc belucid yup
2011-01-20 12:55:38 utc belucid the waiting is not so much where I'm getting stuck
2011-01-20 12:55:51 utc belucid it's the letting ruote know that the work has been performed
2011-01-20 12:56:09 utc jmettraux engine.receive(workitem)
2011-01-20 12:57:16 utc belucid k... so what's the right way to wait for that receive? I was playing with listen :to
2011-01-20 12:57:50 utc belucid but I don't think that's necessarily right
2011-01-20 12:58:17 utc jmettraux the usual way if emit via a participant, let the participant wait
2011-01-20 12:58:37 utc jmettraux listen is mostly for interprocess communication
2011-01-20 12:58:47 utc belucid ok
2011-01-20 12:58:54 utc belucid so, then 2 questions
2011-01-20 12:59:39 utc belucid (not sure why I've gotten so confused on this, since it's so straight forward, but I will re-review the docs and let you know so maybe we can improve them)
2011-01-20 12:59:44 utc jmettraux in context : https://github.com/jmettraux/ruote-amqp/blob/ruote2.1/lib/ruote-amqp/receiver.rb#L110
2011-01-20 13:01:30 utc belucid question 1, is there a way to look up the workitem by wfid? so I can provide it back to receive?
2011-01-20 13:01:32 utc jmettraux yes, the doc is very thin on thid
2011-01-20 13:01:34 utc jmettraux this
2011-01-20 13:01:49 utc belucid I don't want to have to persist the JSON myself
2011-01-20 13:02:52 utc jmettraux engine.process(wfid).workitems
2011-01-20 13:03:08 utc belucid cool
2011-01-20 13:03:10 utc jmettraux engine.process(wfid).stored_workitems
2011-01-20 13:03:39 utc belucid question 2 is what's the best type of participant to just wait? a storage participant?
2011-01-20 13:04:06 utc jmettraux that's hard to answer
2011-01-20 13:04:29 utc jmettraux ok, let me finish answering question 1
2011-01-20 13:04:32 utc belucid sure
2011-01-20 13:05:22 utc jmettraux .workitems : https://github.com/jmettraux/ruote/blob/ruote2.1/lib/ruote/engine/process_status.rb#L213-230
2011-01-20 13:05:50 utc jmettraux .stored_workitems : https://github.com/jmettraux/ruote/blob/ruote2.1/lib/ruote/engine/process_status.rb#L41-46
2011-01-20 13:06:03 utc jmettraux these are the workitems for a given process status
2011-01-20 13:06:08 utc jmettraux now
2011-01-20 13:06:32 utc jmettraux the flow is usually, the participant implementation receives a workitem and emits it entirely or partially "outside"
2011-01-20 13:06:54 utc jmettraux there should at least be the "fei" passed outside or some "correlation id" computed
2011-01-20 13:07:33 utc jmettraux so that when the workitem comes back, the participant (or the receiver) may reconstitute (or fetch back) the workitem, in the ruote sense
2011-01-20 13:07:52 utc jmettraux a quick way to have the correlation id : workitem.fei.sid
2011-01-20 13:07:57 utc jmettraux generates a string
2011-01-20 13:08:32 utc belucid ok... that's exactly what I want to pass out, just a "correlation id"
2011-01-20 13:09:45 utc jmettraux you don't need to get the whole process_status, you can call
2011-01-20 13:09:46 utc jmettraux https://github.com/jmettraux/ruote/blob/ruote2.1/lib/ruote/engine/process_status.rb#L41-46
2011-01-20 13:10:18 utc jmettraux like this : wi = engine.fexp(fei).applied_workitem
2011-01-20 13:10:53 utc jmettraux let me add a engine.workitem(fei) method, I wanted to do it since last year
2011-01-20 13:11:07 utc belucid k
2011-01-20 13:11:31 utc belucid that's exactly what I was looking for yesterday when I conceived of question #1 :-)
2011-01-20 13:12:46 utc belucid also... a 3rd question... is there a repo for the nanoc site/docs? so that I can fork it and help on docs?
2011-01-20 13:13:51 utc jmettraux https://github.com/jmettraux/ruote_website but it's being reworked these days
2011-01-20 13:14:00 utc jmettraux I'm trying to organize stuff
2011-01-20 13:15:43 utc belucid yeah, I've noticed all the changes, everyday I wake up and it's a bit better than the day before
2011-01-20 13:15:45 utc belucid nice
2011-01-20 13:17:15 utc jmettraux :-)
2011-01-20 13:24:41 utc belucid I'm trying things out btw, w/ a storage participant as the "waiter"
2011-01-20 13:24:57 utc jmettraux nice
2011-01-20 13:26:42 utc jmettraux in fact, engine.workitem(fei) is already implemented
2011-01-20 13:27:07 utc jmettraux lacks a test though
2011-01-20 13:27:23 utc belucid oh yeah... and it's for Oleg
2011-01-20 13:27:24 utc belucid :-)
2011-01-20 13:28:38 utc jmettraux adding test and moving it from lib/ruote/engine.rb to lib/ruote/receiver/base.rb
2011-01-20 13:33:44 utc jmettraux https://github.com/jmettraux/ruote/commit/50b45d028d0c2b69a03dbb6bd1cfac5657df67b2 done
2011-01-20 13:37:03 utc belucid nice
2011-01-20 13:37:35 utc belucid any other thoughts on #2 (I'm proceeding down the StorageParticipant path
2011-01-20 13:37:56 utc jmettraux let me re-read
2011-01-20 13:37:59 utc belucid k
2011-01-20 13:38:12 utc jmettraux ok
2011-01-20 13:38:35 utc jmettraux it depends on how you communicate with the "real participant"
2011-01-20 13:39:27 utc jmettraux if you look at the participant implementation given by ruote-amqp
2011-01-20 13:39:41 utc jmettraux it only replies to the engine immediately if "forget" is on
2011-01-20 13:40:27 utc jmettraux in that case the receiver expects to receive the whole workitem back
2011-01-20 13:40:53 utc jmettraux but it's easy to change that so that the workitem is fetched back from the engine and merged with the info that came back
2011-01-20 13:41:40 utc jmettraux extending Ruote::StorageParticipant could be interesting if you want an easy access to all the workitems that are waiting
2011-01-20 13:41:53 utc jmettraux engine.storage_participant.each do |wi| p wi; end
2011-01-20 13:42:02 utc belucid here's a small fragment of the process def
2011-01-20 13:42:09 utc belucid to show you what I'm trying to accomplish:
2011-01-20 13:42:10 utc belucid https://gist.github.com/7dfbc804362826ab30ec
2011-01-20 13:42:40 utc belucid manual_phone_call and complete_as are AMQP participants
2011-01-20 13:43:20 utc belucid once manual_phone_call is done with it, it's been queued up out at Amazon... but it could be hours, minutes, days, weeks, whatever... before the phone call is placed
2011-01-20 13:43:32 utc jmettraux nice
2011-01-20 13:43:46 utc belucid so... I really just want wait_for_phone_call to do nothing
2011-01-20 13:43:53 utc belucid other than block ruote
2011-01-20 13:44:01 utc jmettraux you can write "sequence :if => '${x} == yes' do"
2011-01-20 13:44:17 utc belucid cool
2011-01-20 13:44:39 utc belucid I'll clean that up... in fact... let me post the whole thing in case you see other more idiomatic cleanups
2011-01-20 13:44:55 utc jmettraux so, there are two phone sessions in the process fragment ?
2011-01-20 13:45:17 utc belucid you can refresh that gist
2011-01-20 13:45:20 utc belucid I edited it
2011-01-20 13:45:29 utc jmettraux post the URI over private chat if you don't want to pass it to everyone here
2011-01-20 13:45:36 utc belucid it's fine
2011-01-20 13:45:44 utc belucid ignore line 13
2011-01-20 13:45:56 utc belucid it was just my test of what we are working on
2011-01-20 13:46:30 utc belucid I'll change all the _if :test => sequence dos
2011-01-20 13:47:03 utc jmettraux it looks good, the :if => trick will save you depth
2011-01-20 13:47:12 utc belucid k
2011-01-20 13:47:17 utc belucid line 24, line 35
2011-01-20 13:47:24 utc belucid those look right and idiomatic?
2011-01-20 13:47:33 utc belucid that's how I complete a process early?
2011-01-20 13:47:54 utc jmettraux that's one way
2011-01-20 13:47:56 utc jmettraux looks good
2011-01-20 13:48:00 utc belucid k
2011-01-20 13:48:54 utc belucid so... I'm obviously trying here to avoid introducing a new participant... I could write one that's for mTurk... and collapse manual_phone_call and wait_for_phone_call into 1 participant
2011-01-20 13:48:57 utc jmettraux manual_phone_call and wait_for_phone_call are two different phone sessions ?
2011-01-20 13:49:02 utc belucid no
2011-01-20 13:49:14 utc jmettraux so yes, collapse, well spot
2011-01-20 13:49:17 utc belucid manual_phone_call queues up the job at Amazon
2011-01-20 13:49:34 utc belucid but it's a regular AMQP participant
2011-01-20 13:49:39 utc belucid so as soon as it's done
2011-01-20 13:49:48 utc belucid it'll go on
2011-01-20 13:49:54 utc belucid whereas, we need it to block
2011-01-20 13:50:12 utc belucid until the work actually gets done
2011-01-20 13:50:31 utc belucid so wait_for_phone_call is new... it's what I am trying now
2011-01-20 13:50:39 utc belucid it replaced listen
2011-01-20 13:51:19 utc belucid Its where I'm trying to block ruote until I get a call back from Amazon in my Rails app saying the work is done, and I'll look up the workitem by fei and call receive
2011-01-20 13:51:24 utc belucid does that make sense?
2011-01-20 13:51:29 utc belucid seem like a good approach?
2011-01-20 13:52:25 utc jmettraux I'd rename manual_phone_call to something like queue_up_job
2011-01-20 13:52:36 utc jmettraux and then have a phone_call participant
2011-01-20 13:53:05 utc jmettraux that replies/receives when the phone call is done
2011-01-20 13:53:20 utc belucid ok... that makes sense... to be more clear about the naming and what's happening at each step
2011-01-20 13:53:49 utc belucid but the general approach of doing that in 2 existing participants (AMQP and then Storage)
2011-01-20 13:53:55 utc belucid rather than 1 new custom participant
2011-01-20 13:53:59 utc belucid seems sound?
2011-01-20 13:54:11 utc jmettraux if you have queueing with phone call yes
2011-01-20 13:54:23 utc jmettraux if you have queueing without phone calls yes
2011-01-20 13:54:26 utc jmettraux (sorry)
2011-01-20 13:54:37 utc jmettraux modular
2011-01-20 13:55:03 utc jmettraux you can reuse the queue_up_job in other process definitions
2011-01-20 13:55:39 utc belucid yeah, I agree, queue up job is probably something like this:
2011-01-20 13:56:34 utc belucid queue_for_mechanical_turk :url_callback => :call_back_method_for_this, :pay => 0.25
2011-01-20 13:57:07 utc belucid need to give it a method that can construct the right callback URL and the amount we will pay for the job
2011-01-20 13:59:06 utc jmettraux maybe you don't need to spill this over the process definition, it can stay between the participant implementation and the workitem
2011-01-20 13:59:28 utc belucid spill what over in particular?
2011-01-20 13:59:58 utc jmettraux :url_callback => :call_back_method_for_this
2011-01-20 14:00:12 utc belucid well
2011-01-20 14:00:40 utc belucid yeah... I made it more generic... 1 sec, editing the gist
2011-01-20 14:04:46 utc belucid alright
2011-01-20 14:04:50 utc belucid you can refresh
2011-01-20 14:05:00 utc belucid cleaned up the conditional/sequence syntax
2011-01-20 14:05:25 utc belucid and have the new queue_for_mechanical_turk and manual_* pattern on the human jobs
2011-01-20 14:05:30 utc belucid seems pretty clean I think
2011-01-20 14:05:48 utc jmettraux yes
2011-01-20 14:06:01 utc jmettraux "sequence _if" will make the parser choke
2011-01-20 14:06:10 utc jmettraux "sequence :if" it is
2011-01-20 14:06:31 utc belucid oops
2011-01-20 14:06:42 utc jmettraux concise, clean, I like
2011-01-20 14:07:56 utc belucid I do too, thanks... now I just need to finish that test to make sure I can look up the wi by fei and receive it while it's blocked at one of these manual_* storage participants
2011-01-20 14:32:56 utc jmettraux have a good day
2011-01-20 21:30:46 utc wayneeseguin jmettraux when you are around I'd like to see if you'd be interested in that guest post for 02.14.2011 :)
2011-01-21 00:05:20 utc jmettraux wayneeseguin: february the 14th, it sounds OK, 2 weeks... It means I have to be concise, which is not a bad thing