2012-11-01 00:59:11 utc |
mburnett |
i got my little experiment with killing workers to work, i was also killing associated participant (sleepers) and hadn't setup timeouts |
2012-11-01 01:08:06 utc |
jmettraux |
oh, cool |
2012-11-01 01:09:36 utc |
mburnett |
yeah, this is a great library..also, I really like that you wrote it as a library rather than a direct service |
2012-11-01 01:24:48 utc |
jmettraux |
trying to not get too much in the way... |
2012-11-01 09:27:47 utc |
jmettraux |
mburnett: about submitting jobs to a grid engine and waiting for them to complete |
2012-11-01 09:28:48 utc |
jmettraux |
maybe looking at ruote-amqp and how it features participants/listeners might help (https://github.com/kennethkalmer/ruote-amqp) |
2012-11-01 09:29:08 utc |
jmettraux |
s/listener/receiver/ |
2012-11-01 09:36:34 utc |
jmettraux |
basically the participant is signing the outgoing "workitem" with the string version of its "fei" (flow expression id), and the external system is supposed, when it replies, to include that identifier |
2012-11-01 09:36:58 utc |
jmettraux |
so that the receiver, in the engine, knows what expression / workflow instance to resume |
2012-11-01 20:56:49 utc |
kindjal |
Trying hard to find a working example of ruote + amqp + rabbit... |
2012-11-01 20:56:55 utc |
kindjal |
finding lots of examples that don't work |
2012-11-01 20:57:09 utc |
kindjal |
or say "ruote-amqp" but mean "RuoteAMQP" which seem different |
2012-11-01 20:57:36 utc |
kindjal |
Any hints? |
2012-11-01 20:58:14 utc |
phaeron |
we have a working solution with ruote-amqp and rabbitmq and python participant |
2012-11-01 20:58:15 utc |
phaeron |
s |
2012-11-01 20:58:26 utc |
kindjal |
is it on github? |
2012-11-01 20:58:51 utc |
phaeron |
lbt: ping where is the last place you put boss-bundle |
2012-11-01 20:58:58 utc |
phaeron |
kindjal: will rpm packages do ? |
2012-11-01 20:59:03 utc |
kindjal |
sure |
2012-11-01 20:59:05 utc |
lbt |
umm |
2012-11-01 20:59:39 utc |
phaeron |
https://build.pub.meego.com/project/packages?project=Project:MINT:Testing |
2012-11-01 20:59:47 utc |
phaeron |
kindjal: boss and boss-bundle there |
2012-11-01 20:59:51 utc |
lbt |
https://build.pub.meego.com/package/files?package=webapp-bundle&project=home%3Albt%3Abranches%3AMer%3ATools%3ATesting |
2012-11-01 20:59:57 utc |
phaeron |
it's underdocumented unforunately |
2012-11-01 21:00:06 utc |
kindjal |
phaeron: I'll take a look. Thanks! |
2012-11-01 21:00:22 utc |
phaeron |
welcome |
2012-11-01 21:00:30 utc |
phaeron |
I gtg now but I will login later |
2012-11-01 21:00:52 utc |
mburnett |
jmettraux: thanks for the suggestion about amqp, i'm looking into it now |
2012-11-01 21:01:07 utc |
jmettraux |
kindjal: for ruote-amqp examples that work, I know for sure that the ruote-amqp specs do pass, so those specs (and the README) should be OK |
2012-11-01 21:01:14 utc |
lbt |
https://build.pub.meego.com/package/view_file?file=update-pkg-bundle.sh&package=sdk-webapp-bundle&project=home%3Albt%3Abranches%3AMer%3ATools%3ATesting&rev=efcf5fd6dcf73064028b4ebeb7e958ca |
2012-11-01 21:01:15 utc |
jmettraux |
mburnett: you're welcome |
2012-11-01 21:02:03 utc |
lbt |
mmm - didnt' read the surrounding discussion either |
2012-11-01 21:02:16 utc |
kindjal |
jmettraux: I should just read the ./spec, shouldn't I…. I'll do that too. |
2012-11-01 21:02:39 utc |
jmettraux |
kindjal: the ruote-amqp readme and then the spec |
2012-11-01 21:03:28 utc |
kindjal |
jmettraux: Yes, I've been over the read me… but being new to ruote, and not very experienced with Ruby at all, it's difficult to put together how it all works to build something new. |
2012-11-01 21:03:32 utc |
jmettraux |
and if you want to make a particular example "work", we can help, but you'd need to tell us which example |
2012-11-01 21:03:43 utc |
kindjal |
I'll get there. |
2012-11-01 21:04:18 utc |
jmettraux |
I was afraid of that, people learning wide instead of deep and then labelling it all as "not working" :-( |
2012-11-01 21:05:00 utc |
jmettraux |
also if you spot issues, we can fix them |
2012-11-01 21:06:05 utc |
kindjal |
I'm sorry, I really am doing that aren't I. |
2012-11-01 21:06:06 utc |
kindjal |
My fault |
2012-11-01 21:06:10 utc |
lbt |
as a new-to-ruby guy I had problems getting up to speed with it all; I recall thinking some of the examples were a triumph of ruby-minimalism over clarity of understanding :) |
2012-11-01 21:06:28 utc |
jmettraux |
lol |
2012-11-01 21:06:40 utc |
lbt |
idoms should be banned in tutorials :) |
2012-11-01 21:07:02 utc |
jmettraux |
the next step is to ban them from all code |
2012-11-01 21:07:03 utc |
lbt |
or mentioned as elegant usage styles |
2012-11-01 21:07:20 utc |
lbt |
I love them in real code |
2012-11-01 21:07:23 utc |
kindjal |
phaeron left… but I see his "boss" app uses RuoteAMQP, which is not the same as Ruote::Amqp is it? |
2012-11-01 21:07:39 utc |
lbt |
kindjal: phaeron and I work together |
2012-11-01 21:07:45 utc |
jmettraux |
+1 for commented out "this code, in layman Ruby, would look like" |
2012-11-01 21:08:03 utc |
kindjal |
lbt: what's the story with RuoteAMQP? |
2012-11-01 21:08:07 utc |
lbt |
kindjal: python |
2012-11-01 21:08:21 utc |
kindjal |
but boss is #!/usr/bin/ruby? |
2012-11-01 21:08:28 utc |
lbt |
*g* |
2012-11-01 21:08:39 utc |
lbt |
that's standard protocols for you |
2012-11-01 21:09:01 utc |
lbt |
so we have boss/ruby talk over amqp to participants/python |
2012-11-01 21:09:18 utc |
lbt |
we choose to do that btw |
2012-11-01 21:09:35 utc |
kindjal |
umm |
2012-11-01 21:09:36 utc |
kindjal |
http://ruote.rubyforge.org/ruote-amqp_rdoc/RuoteAMQP.html |
2012-11-01 21:09:36 utc |
kindjal |
? |
2012-11-01 21:09:54 utc |
kindjal |
which is not https://github.com/kennethkalmer/ruote-amqp |
2012-11-01 21:09:55 utc |
jmettraux |
argh, out of date rdoc, I should simply nuke it |
2012-11-01 21:10:25 utc |
kindjal |
I am so confused |
2012-11-01 21:10:30 utc |
lbt |
oops - sorry, my mistake |
2012-11-01 21:10:44 utc |
kindjal |
eg: RuoteAMQP::Receiver.new( $engine, :launchitems => true ) |
2012-11-01 21:10:51 utc |
kindjal |
that is not "how it is done" with ruote-amqp |
2012-11-01 21:10:55 utc |
jmettraux |
sorry, this is all outdated |
2012-11-01 21:11:05 utc |
jmettraux |
https://github.com/kennethkalmer/ruote-amqp is the "source" |
2012-11-01 21:11:19 utc |
kindjal |
I so want a working example that makes use of that. |
2012-11-01 21:11:30 utc |
jmettraux |
listed via http://ruote.rubyforge.org/source.html |
2012-11-01 21:12:15 utc |
kindjal |
yes, that's how I got to ruote-amqp, but then you go googling for "ruote ruote-amqp examples" and you get a bunch of github projects that say "ruote-amqp" but actually use RuoteAMQP |
2012-11-01 21:12:37 utc |
kindjal |
I'll just try the spec examples and read me from ruote-amqp |
2012-11-01 21:13:03 utc |
kindjal |
btw, I'm super excited about ruote, and yes, I immediately jumped into a complex example I want to build... |
2012-11-01 21:13:10 utc |
kindjal |
just want to say "this looks so cool!" |
2012-11-01 21:13:10 utc |
kindjal |
;) |
2012-11-01 21:13:42 utc |
jmettraux |
was looking at http://jmettraux.wordpress.com/2010/09/27/amqp-ruote-pong/ but the example by Hartog it points out is "not responding" :-( |
2012-11-01 21:14:45 utc |
jmettraux |
ACTION adds to his todo: "write pong example for ruote-amqp" |
2012-11-01 21:17:41 utc |
jmettraux |
lbt: if you come across those "too elegant to be readable" examples, please fire an issue report ;-) |
2012-11-01 21:17:50 utc |
lbt |
will do :) |
2012-11-01 21:17:53 utc |
lbt |
and kindjal: apologies for confusing you too |
2012-11-01 21:18:06 utc |
kindjal |
lbt: no worries |
2012-11-01 21:18:38 utc |
lbt |
I will vouch for ruote over amqp as being a great solution |
2012-11-01 21:19:02 utc |
kindjal |
that's what my gut tells me… I just need to make it work! ;) |
2012-11-01 21:48:18 utc |
kindjal |
if a participant launches a grid job, say via SGE, and the job takes days, how does ruote know the job's health... if it dies, we want to know, if it's complete, we want to progress. |
2012-11-01 21:52:14 utc |
lbt |
so the participant takes the jobdesc from amqp; launches the 'grid job' and stores the workitem. it then exits |
2012-11-01 21:52:40 utc |
lbt |
when your job is done the workitem is loaded and sent back to ruote |
2012-11-01 21:52:58 utc |
lbt |
if you reboot the client and the job restarts, that's fine |
2012-11-01 21:53:31 utc |
lbt |
that's one way you could do it |
2012-11-01 21:53:46 utc |
lbt |
the "job is done" is up to you |
2012-11-01 21:54:03 utc |
kindjal |
so the grid job would hold the responsibility to say "I'm done"... |
2012-11-01 21:54:17 utc |
kindjal |
but if it died, how would ruote know, so that the workflow could restart it? |
2012-11-01 21:54:23 utc |
kindjal |
it'd have to have a heartbeat monitor. |
2012-11-01 21:54:24 utc |
lbt |
you're asking the wrong way round |
2012-11-01 21:54:42 utc |
lbt |
you don't design your systems around ruote as such |
2012-11-01 21:54:59 utc |
lbt |
you design your systems and use ruote to communicate |
2012-11-01 21:55:16 utc |
lbt |
if you would like a heartbeat for other reasons, then have one |
2012-11-01 21:55:29 utc |
kindjal |
I guess I'm trying to learn what ruote "does by itself"... |
2012-11-01 21:56:12 utc |
lbt |
it asks systems to do things and enables you to flexibly do things based on the results |
2012-11-01 21:57:24 utc |
lbt |
eg we use it to close bugs when QA tests pass if the changelog mentions "FIXES" a bug and it's built in a certain project |
2012-11-01 21:57:57 utc |
kindjal |
I think I understand what you're saying... |
2012-11-01 21:58:05 utc |
kindjal |
I think. |
2012-11-01 21:58:07 utc |
kindjal |
:) |
2012-11-01 21:58:34 utc |
kindjal |
If I have remote processes as 'workers', I'll need to know when they die, or error, or complete... |
2012-11-01 21:58:48 utc |
kindjal |
and I think I'll have to build these bits… and then communicate via ruote to "do the right thing" |
2012-11-01 21:58:58 utc |
kindjal |
I think |
2012-11-01 21:59:03 utc |
kindjal |
I'll go work on it |
2012-11-01 21:59:05 utc |
kindjal |
thanks. |
2012-11-01 21:59:26 utc |
lbt |
ping me if you want to bounce ideas about |
2012-11-01 21:59:35 utc |
kindjal |
Thanks, I probably wil.. |
2012-11-01 21:59:43 utc |
lbt |
I tend to use ruote with multiple servers running different tasks |
2012-11-01 22:00:01 utc |
lbt |
rather than small sw components/functions |
2012-11-01 22:01:35 utc |
lbt |
so 'build an rpm' on one server; create an installable image on another; run tests on devices using that image on another; update DB according to results; notifications for success/fail |
2012-11-01 22:02:33 utc |
lbt |
if the image builder goes down it's designed to come up and continue with it's own queue (image can take many minutest to build) |
2012-11-01 22:03:06 utc |
lbt |
the db updated is more RPC - transaction is pretty instant |
2012-11-01 22:03:58 utc |
kindjal |
I follow you. |
2012-11-01 22:04:06 utc |
lbt |
we're not quite at the : pull from amqp; do work; ack message+respond atomically- AMQP flow yet (I don't think) |
2012-11-01 22:04:51 utc |
lbt |
also we did some stress work at Nokia a year or so ago |
2012-11-01 22:05:09 utc |
lbt |
and ran millions of processes or something |
2012-11-01 22:05:24 utc |
lbt |
found some minor mem leaks but in general all went well |
2012-11-01 22:05:53 utc |
lbt |
we need to fixup the "admin interface" which can be a pita |
2012-11-01 22:06:22 utc |
lbt |
join #meego-boss btw |
2012-11-01 22:06:38 utc |
lbt |
we have an irc logger which is sometimes interesting |
2012-11-01 22:08:58 utc |
lbt |
this is one of our more complex processes: http://autodoc.meego.com/boss/processes/CE/Adaptation/N900/SRCSRV_REQUEST_CREATE.CE.pdef |
2012-11-01 22:09:50 utc |
lbt |
we don't like jmettraux to see what twisted things we do with his baby though ... |
2012-11-01 22:15:04 utc |
phaeron |
I showed him that one before and he said it was ok :D |
2012-11-01 22:17:24 utc |
kindjal |
Thanks for the tips. I'm going to go dig into it more. ;) |
2012-11-01 22:49:38 utc |
jmettraux |
ok #meego-boss is super noisy |
2012-11-01 22:50:10 utc |
jmettraux |
just hoping ruote specific conversation stay here |
2012-11-01 22:50:22 utc |
jmettraux |
lbt thanks so much for your help here :-) |
2012-11-01 22:52:40 utc |
phaeron |
yes it reports on every event that happens in the build system |
2012-11-01 22:54:09 utc |
lbt |
jmettraux: np - the #meego-boss is interesting every now and again: http://pastie.org/5169149 |
2012-11-01 23:49:58 utc |
jmettraux |
cool stuff |