2013-01-16 00:27:21 utc |
jmettraux |
side note, just wrote that: http://jmettraux.github.com/2013-01-16-neg.html sorry for the noise |
2013-01-16 01:11:23 utc |
randomcamel |
how extensible would you guys say Ruote is? we have our own systems for distributing and scheduling jobs out to workers, and I'm looking at whether it's possible to integrate Ruote with those things. I guess what would be needed would be callbacks or hooks to extend or replace Ruote events. |
2013-01-16 01:12:02 utc |
jmettraux |
why not simply ruote participants and receivers that interface with your system? |
2013-01-16 01:13:03 utc |
jmettraux |
à la https://github.com/kennethkalmer/ruote-amqp or https://github.com/jmettraux/ruote-beanstalk/ |
2013-01-16 01:18:05 utc |
randomcamel |
when I looked into that a while ago, I couldn't figure out where I would implement more sophisticated scheduling (e.g. arbitrary priorities or limited resources). |
2013-01-16 01:19:24 utc |
randomcamel |
(I didn't look at either of those two things, but that was one of the fundamental problems with using any of the FIFO work queues.) |
2013-01-16 01:20:57 utc |
jmettraux |
seems like you already have a level of indirection |
2013-01-16 01:21:36 utc |
jmettraux |
I guess a ruote participant would just have to know which of your work queue and how to flag a task with a priority |
2013-01-16 01:22:12 utc |
jmettraux |
probably the priority label is set by default in the participant and can be overriden via the workitem or the process definition |
2013-01-16 01:22:28 utc |
jmettraux |
sorry, just letting my imagination loose, I have no idea of your context |
2013-01-16 01:30:34 utc |
randomcamel |
no worries, I like ideas. =) it's video encoding and packaging workflows. |
2013-01-16 01:30:52 utc |
jmettraux |
sweet :-) |
2013-01-16 01:32:03 utc |
randomcamel |
we've been working on other aspects and putting off the nuts and bolts of how we describe workflows (which basically have to be dynamically generated from an existing kind of specification) and realizing I should have thought about it harder. |
2013-01-16 01:33:25 utc |
randomcamel |
remembering that I liked Ruote's DSL but I was really really wary (and still am) of deep integration with and modification of existing job distribution systems, since IME they all have their own very strong ideas of how the world works and it's just as much work to conform to the tool's worldview as it is to write a basic version yourself. |
2013-01-16 01:34:22 utc |
jmettraux |
understood |
2013-01-16 01:34:29 utc |
randomcamel |
but coming up now to what we actually need to express in our workflows, I'm wondering if I should have involved Ruote from the beginning, and if it's possible to do it now. |
2013-01-16 01:37:23 utc |
jmettraux |
if ruote can "compose" with your system, ie if your system present a well defined {sur|inter}face, I'm sure it can |
2013-01-16 01:37:46 utc |
jmettraux |
at least that would be invaluable in order to develop your own orchestration component |
2013-01-16 01:37:52 utc |
jmettraux |
ruote or not |
2013-01-16 09:57:53 utc |
hartog |
hi all... |
2013-01-16 10:00:16 utc |
mister_solo |
good morning |
2013-01-16 10:06:52 utc |
hartog |
Does anyone here has any suggestions? http://stackoverflow.com/questions/14354287/mongo-vs-mongoid-why-can-1-connect-and-the-other-not |
2013-01-16 10:07:28 utc |
jmettraux |
hartog: hello |
2013-01-16 10:08:04 utc |
jmettraux |
please look at https://jira.mongodb.org/browse/RUBY-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel |
2013-01-16 10:09:54 utc |
jmettraux |
you should mention platforms and versions when you report issues like that |
2013-01-16 10:09:57 utc |
hartog |
1.8.0, 1.8.1, 1.9.0, etc. refer to gem version? |
2013-01-16 10:10:17 utc |
jmettraux |
yes |
2013-01-16 10:10:34 utc |
jmettraux |
from what I remember from your setting, you were already using ruote-mon |
2013-01-16 10:10:37 utc |
jmettraux |
so it worked |
2013-01-16 10:11:07 utc |
jmettraux |
at some point you upgraded to mongo 1.8.1 and it stopped working |
2013-01-16 10:11:37 utc |
jmettraux |
do not be afraid to look at your version control diffs, they have lots of info for you |
2013-01-16 10:11:42 utc |
hartog |
unwillingly, unknowingly and undocumented - story of ruby |
2013-01-16 10:11:47 utc |
jmettraux |
proceed with small steps |
2013-01-16 10:12:12 utc |
hartog |
problem is, I did not upgrade |
2013-01-16 10:12:25 utc |
hartog |
but my mongo gem is not pinned down |
2013-01-16 10:12:28 utc |
hartog |
so I will make it so |
2013-01-16 10:12:39 utc |
jmettraux |
your Gemfile.lock is pinning it |
2013-01-16 10:12:54 utc |
jmettraux |
be a bit more methodical and your life will be easier |
2013-01-16 10:14:03 utc |
hartog |
I am pretty methodical |
2013-01-16 10:14:19 utc |
hartog |
and as I suspected; the version of mongo (1.8.1) did not change since jan. 9 |
2013-01-16 10:14:29 utc |
hartog |
on my development machine it works, on production it does not |
2013-01-16 10:14:36 utc |
jmettraux |
what OS? |
2013-01-16 10:14:40 utc |
hartog |
both carry the same (boasted) mongod version |
2013-01-16 10:14:43 utc |
jmettraux |
what OSes |
2013-01-16 10:14:50 utc |
hartog |
dev=ubuntu, prod=debian |
2013-01-16 10:15:12 utc |
jmettraux |
are you sure it's mongo 1.8.1 on both? |
2013-01-16 10:15:30 utc |
hartog |
both have the same gemfile lock |
2013-01-16 10:15:50 utc |
hartog |
down to the md5sum |
2013-01-16 10:18:18 utc |
jmettraux |
ok it's solved, it would have been interesting to know what difference(s) lead to the issue |
2013-01-16 10:18:32 utc |
hartog |
i will take a source build mongo to see what happens - thx for your help and pointers |
2013-01-16 10:19:06 utc |
jmettraux |
you're welcome |
2013-01-16 10:24:31 utc |
hartog |
and voila; it works |
2013-01-16 10:24:34 utc |
hartog |
when I die, I will be placed in that circle of hell where you have to solve dependency problems all day |
2013-01-16 10:31:10 utc |
jmettraux |
:-) |
2013-01-16 10:31:38 utc |
jmettraux |
better better than being reincarnated as a goat or as a windows sysadmin |
2013-01-16 10:33:09 utc |
hartog |
I would love to be reincarnated as a goat - just eat, sleep, head-butt-your-peers and climb-stuff all day. Seems perfect to me... :D |
2013-01-16 10:33:50 utc |
jmettraux |
:D, yeah, well said |
2013-01-16 13:00:47 utc |
hartog |
jmettraux: as an extension to 'testing participants' on the ruote website: https://gist.github.com/4546918 |
2013-01-16 13:06:51 utc |
jmettraux |
ok, thanks, I'm still left with writing the documentation |
2013-01-16 13:07:03 utc |
jmettraux |
as I said, people prefer to write code ;-) |
2013-01-16 13:09:22 utc |
hartog |
i know... |
2013-01-16 13:09:33 utc |
hartog |
I will add more comments as I further distill this |
2013-01-16 13:09:55 utc |
jmettraux |
adding that to my todo list |
2013-01-16 13:10:00 utc |
hartog |
btw; there is nothing in there that is not documented (except for the thingy with the fei) |
2013-01-16 13:10:04 utc |
hartog |
you just have to look hard |
2013-01-16 13:11:06 utc |
jmettraux |
yeah, that's the trick, people complain about the documentation enough that I'm motivated to write it |
2013-01-16 13:12:34 utc |
jmettraux |
for now I linked to your gist: http://ruote.rubyforge.org/testing_participants.html |
2013-01-16 13:12:57 utc |
jmettraux |
I'll try to find so time during the week-end to refine your work into some documentation |
2013-01-16 13:24:00 utc |
hartog |
i'll see if i can write a few words so you can dress it up further - I should have some time some where |
2013-01-16 13:29:42 utc |
jmettraux |
have to go, thanks and thanks for bearing my grumpiness ;-) |
2013-01-16 13:58:28 utc |
hartog |
john; I know you scour the IRC logs from time to time - I added a little readme to https://gist.github.com/4546918, hope it helps ;) |
2013-01-16 14:26:24 utc |
MCamou |
hi all |
2013-01-16 14:26:35 utc |
MCamou |
I am looking at Hartog's participant testing gist |
2013-01-16 14:26:47 utc |
MCamou |
which comes at just the right time for us as we are just getting into testing :) |
2013-01-16 14:26:51 utc |
MCamou |
quick question |
2013-01-16 14:27:12 utc |
MCamou |
if a participant sets a variable, how can I check its value in a test? |
2013-01-16 14:27:51 utc |
MCamou |
we're using fexp.set_variable inside our participant |
2013-01-16 14:29:19 utc |
hartog |
ACTION is clueless :D |
2013-01-16 14:30:17 utc |
hartog |
john suggested once to steer clear from interacting with variables (setting, etc) and try to keep it within the workitem. I try to live by that pragma... |
2013-01-16 14:30:35 utc |
hartog |
but you could use mocking for testing the set_variable has been called |
2013-01-16 14:31:06 utc |
MCamou |
hmmmm… I didn't get the memo re: variables :) |
2013-01-16 14:31:32 utc |
MCamou |
but I'll try mocking |
2013-01-16 14:31:34 utc |
hartog |
with mocha something like: ```subject.fexp.expects(:set_variable)``` |
2013-01-16 14:31:50 utc |
hartog |
variables are engine_variables |
2013-01-16 14:32:02 utc |
hartog |
afai[k/c] |
2013-01-16 14:32:26 utc |
hartog |
workitem.fields are workflow variables, so to speak |
2013-01-16 14:32:50 utc |
MCamou |
right but we see variables as something a bit less permanent |
2013-01-16 14:34:09 utc |
hartog |
you have my blessings on the issue - I try to understand ruote, but it is though-stuff |
2013-01-16 14:34:25 utc |
hartog |
so I cannot for the world give you any good advise on how (not) to use variables |
2013-01-16 14:35:38 utc |
MCamou |
hartog: thanks, using mocks should help a lot :) |
2013-01-16 14:36:45 utc |
MCamou |
BTW, small typo. In your gist you're calling the module RuoteHelpers and calling config.include RuoteHelper (without the 's') |
2013-01-16 14:37:03 utc |
hartog |
okthx! |
2013-01-16 14:37:36 utc |
hartog |
fixed :) |
2013-01-16 14:39:03 utc |
MCamou |
oh another one :) in the test you require 'spec_helper' and the file is named ruote_helper.rb |
2013-01-16 14:39:45 utc |
MCamou |
and of course we're missing a require 'ruote' or something |
2013-01-16 14:39:57 utc |
MCamou |
or Ruote::LocalParticipant won't be found |
2013-01-16 14:41:22 utc |
hartog |
but obviously; it is a hook for you to base your work on |
2013-01-16 14:41:31 utc |
MCamou |
yes of course :) |
2013-01-16 14:41:36 utc |
MCamou |
thanks for your great work! |
2013-01-16 14:41:50 utc |
hartog |
normally when you use rspec, there would/should be a spec/ and spec/spec_helper.rb file |
2013-01-16 14:42:01 utc |
hartog |
np |
2013-01-16 15:13:20 utc |
hartog |
ACTION is off |
2013-01-16 15:13:24 utc |
hartog |
wave! |
2013-01-16 15:15:04 utc |
MCamou |
hartog: one more thing… in your RuoteHelpers#new_workitem, merge should be merge! |
2013-01-16 21:21:10 utc |
randomcamel |
so I'm a little confused about what Ruote::Worker actually does. in my mind a "worker" is the thing that actually does the task work, but AFAICT in Ruote that's actually the participant. |
2013-01-16 21:22:12 utc |
randomcamel |
it looks like maybe Worker is some kind of artifact from previous versions? |
2013-01-16 22:28:47 utc |
jmettraux |
randomcamel: hello |
2013-01-16 22:28:59 utc |
jmettraux |
no, worker are not artifacts from previous versions |
2013-01-16 22:29:15 utc |
jmettraux |
the execution of a workflow instance is the sum of little actions |
2013-01-16 22:29:24 utc |
jmettraux |
those little actions are performed by workers |
2013-01-16 22:29:48 utc |
jmettraux |
among those actions are participant actions |
2013-01-16 22:30:16 utc |
jmettraux |
it helps if you say to yourself "ruote worker" |
2013-01-16 22:33:10 utc |
jmettraux |
I chose the name "worker" because it picks work on a queue, performs it and potentially queues more work |
2013-01-16 22:38:33 utc |
randomcamel |
huh, they are sparsely mentioned in the docs. are they picking work off a queue that's done behind the scenes in the storage? |
2013-01-16 22:39:55 utc |
jmettraux |
yes |
2013-01-16 22:44:22 utc |
randomcamel |
so if you were to set up a group of machines to execute process instances, would there be a handful of machines running engine+worker, and then the majority being dumber nodes that are just participants? |
2013-01-16 22:45:02 utc |
jmettraux |
out of the box, the workers run the participants |
2013-01-16 22:45:15 utc |
jmettraux |
so it's a handful of worker machines |
2013-01-16 22:45:42 utc |
jmettraux |
unless your participants are just proxy to another job system |
2013-01-16 22:46:08 utc |
jmettraux |
in that case you'd have the ruote worker machines and the other job system machines |
2013-01-16 22:47:23 utc |
randomcamel |
is the engine a central-ish component, or are all the worker+participant machines also running the engine? |
2013-01-16 22:48:05 utc |
jmettraux |
the Engine is deprecated in favour of the Dashboard in ruote 2.3.x |
2013-01-16 22:48:45 utc |
jmettraux |
it's only a dashboard, it just wraps the storage or the couple worker+storage with methods for querying workflow executions, interrupting them or launching new ones |
2013-01-16 22:49:38 utc |
jmettraux |
all the recent examples should avoid Engine and use Dashboard. If you find an outdated example, please link me to it and I'll fix it if possible |
2013-01-16 22:50:11 utc |
randomcamel |
http://ruote.rubyforge.org/glossary.html =) |
2013-01-16 22:50:30 utc |
randomcamel |
is there an easy way to submit doc patches? |
2013-01-16 22:51:04 utc |
jmettraux |
yes, https://github.com/jmettraux/ruote_website |
2013-01-16 22:51:12 utc |
randomcamel |
ah hah! sweet. |
2013-01-16 22:51:37 utc |
randomcamel |
writing and correcting docs is a good way to learn. |
2013-01-16 22:53:03 utc |
jmettraux |
thanks! http://ruote.rubyforge.org/glossary.html fixed |
2013-01-16 22:57:02 utc |
randomcamel |
so when a participant has finished its action, is it the worker that decides on the next step, based on the process definition? |
2013-01-16 22:57:35 utc |
jmettraux |
seen from above, yes |
2013-01-16 22:57:37 utc |
randomcamel |
(this is all really helpful, by the way. it's a different model and terminology than I've seen before, so I'm trying to map out each things responsibilities.) |
2013-01-16 22:58:01 utc |
jmettraux |
:-) |
2013-01-16 23:03:53 utc |
randomcamel |
I am very interested in not re-implementing this stuff myself. |