ruote tmp/log_2013-01-17.html

2013-01-17 00:19:30 utc ypz hi, john
2013-01-17 00:20:30 utc jmettraux ypz: hi
2013-01-17 00:20:44 utc ypz I am also RTFM to understand route terminology and trying to visualize our processes in such terms
2013-01-17 00:21:34 utc ypz here is question: if I need to run some cmd on a number of nodes concurrently, I would configure one participant for each node, correct ?
2013-01-17 00:22:24 utc jmettraux ok, I'll emit an answer that completely disregards your context, because I know nothing of your context
2013-01-17 00:22:34 utc jmettraux you'd need to have a ruote worker on each node
2013-01-17 00:22:44 utc jmettraux and then each worker would greedily pick work
2013-01-17 00:23:09 utc jmettraux so you wouldn't, out of the box, know which worker would pick the work
2013-01-17 00:24:14 utc jmettraux participants are not mapped to nodes or workers
2013-01-17 00:24:21 utc jmettraux workers instantiate them when needed
2013-01-17 00:27:19 utc ypz Hm I am more confused by your answer :(, that says how little I understood how route works !
2013-01-17 00:27:44 utc ypz here is a more concrete scenario:
2013-01-17 00:27:56 utc jmettraux ok, I'll make it shorter: you cannot map a participant to a node
2013-01-17 00:30:46 utc ypz let's say I have 5 node, I need to correct current uptime on all of them, for a shell script, I'd do : for n in n1 n2 n3 n4 n5; do ssh $n uptime; done
2013-01-17 00:31:09 utc jmettraux ok
2013-01-17 00:31:11 utc ypz how would I map this task into a route process ?
2013-01-17 00:31:32 utc ypz s/correct/collect/
2013-01-17 00:37:55 utc jmettraux ypz: something like: https://gist.github.com/4552359
2013-01-17 00:43:01 utc ypz Here is my understand of reading this: the process contains 5 instances of grab_uptime participant, each one will ssh into a different node and get results
2013-01-17 00:43:17 utc jmettraux yes
2013-01-17 00:43:40 utc ypz the worker runs on my local node, and there is only one worker
2013-01-17 00:43:45 utc jmettraux yes
2013-01-17 00:44:36 utc ypz those 5 instances of the same participant runs concurrently, each in a separate thread, the worker runs in parent thread
2013-01-17 00:44:58 utc jmettraux out of the box, yes
2013-01-17 00:45:15 utc ypz right
2013-01-17 00:48:15 utc ypz if the number of nodes are in hundreds or even thousands, how do I limit number of concurrent threads to a manageable number, say 64, in this scenario?
2013-01-17 00:49:17 utc jmettraux you could a) provide your own implementation of the DispatchPool, it's the component in ruote that manages participant threads
2013-01-17 00:50:03 utc jmettraux b) let your participant implementation respond "true" to the method do_not_thread and have multiple workers, such participants monopolize the worker until done
2013-01-17 00:50:27 utc jmettraux the easiest is probably a)
2013-01-17 00:54:39 utc jmettraux combination of a) and b) is good, especially with a Ruby that has a global interpreter lock
2013-01-17 00:56:48 utc ypz by "having multiple workers" do you mean creating multiple Ruote::Worker instances attached to the same storage instance?
2013-01-17 00:57:33 utc jmettraux yes, different ruby processes, same host or different host
2013-01-17 00:59:07 utc ypz do I need to have multiple dashboard instances as well ?
2013-01-17 00:59:50 utc jmettraux only if you want to interact with the system (the storage and its workers)
2013-01-17 01:00:24 utc jmettraux it's OK to have "nodes" where you only have worker(storage), not the whole dashboard(worker(storage))
2013-01-17 01:00:50 utc jmettraux the dashboard is only a thin layer of commands (indicators and a few steering wheels)
2013-01-17 01:06:52 utc ypz can I attach multiple worker instances with a single dashboard instance so that when I run route.launch)pref), all workers get into action ?
2013-01-17 01:17:33 utc jmettraux yes, they'll compete to get into action
2013-01-17 01:19:34 utc ypz I have not seen much discussion of this workitem queue in the docs, is it only intended for Ruote internal usage ?
2013-01-17 01:19:48 utc jmettraux there is no workitem queue
2013-01-17 01:19:53 utc jmettraux there is a "msg" queue
2013-01-17 01:21:20 utc jmettraux by "instance" I understand "instance of a Ruby object", to make it clear, running multiple worker instances in the same Ruby process won't buy you concurrency
2013-01-17 01:23:28 utc ypz neither this msg nor msg queue is mentioned in glossary :(
2013-01-17 01:27:30 utc jmettraux you normally don't have to bother about it
2013-01-17 01:28:33 utc jmettraux ACTION adds "add 'msg'" to the glossary with a "low level" label
2013-01-17 01:30:41 utc jmettraux the glossary isn't meant to explain low level details
2013-01-17 01:32:54 utc jmettraux other pieces of documentations mention it
2013-01-17 01:33:23 utc ypz ACTION reading rdoc now
2013-01-17 01:37:10 utc jmettraux sorry, my fault, the rdoc output available on the net is probably seriously outdated
2013-01-17 01:39:46 utc ypz hey
2013-01-17 01:41:11 utc jmettraux unless it's on one of those automated rdoc sites
2013-01-17 01:43:06 utc ypz here is what I am reading: http://ruote.rubyforge.org/rdoc/README_rdoc.html
2013-01-17 01:44:13 utc jmettraux yes, sorry, this is outdated
2013-01-17 01:55:02 utc ypz btw, is irc conversations on this channel archived anywhere ?
2013-01-17 01:56:02 utc jmettraux http://ruote-irclogs.s3.amazonaws.com/logs.html
2013-01-17 01:58:35 utc ypz great, I am sure many basic questions have been asked and answered over and over again on this channel !
2013-01-17 01:58:45 utc jmettraux and on the mailing list
2013-01-17 01:59:14 utc jmettraux https://groups.google.com/forum/?fromgroups#!forum/openwferu-users
2013-01-17 01:59:31 utc ypz yea, I am also browsing mailings list too
2013-01-17 01:59:37 utc jmettraux always tried to patch the doc as a follow up to questions on the mailing list or on irc
2013-01-17 02:00:11 utc jmettraux but usually people tend to ask first, with their terminology
2013-01-17 02:00:26 utc jmettraux on irc
2013-01-17 02:06:05 utc ypz john, thanks a lot for you help
2013-01-17 02:06:38 utc ypz got to leave now, bye
2013-01-17 02:06:40 utc jmettraux you're welcome, I'll try to update the rdoc, realizing that rdoc.info is down/gone
2013-01-17 02:06:46 utc jmettraux ok, have a good evening!
2013-01-17 07:22:01 utc hartog hi all!
2013-01-17 08:31:56 utc jmettraux hello!
2013-01-17 08:37:19 utc hartog hello!
2013-01-17 08:38:13 utc jmettraux do I have your permission for integrating https://gist.github.com/4546918 into the ruote website?
2013-01-17 08:38:53 utc hartog yes you have; but it was/is not quite finished yet
2013-01-17 08:39:07 utc hartog I still have to describe why you should use subject._on_workitem, etc.
2013-01-17 08:39:15 utc hartog but if you wish to do it yourself .... ;)
2013-01-17 08:39:18 utc jmettraux ok, then I'll wait
2013-01-17 08:39:33 utc jmettraux no, help with documentation is warmly welcome
2013-01-17 08:40:05 utc hartog i will flush some time in it some time this week (if not today)
2013-01-17 08:40:47 utc jmettraux thanks in advance!
2013-01-17 10:57:39 utc jmettraux MCamou: hello Mario, thanks for the feedback you gave to hartog yesterday, much appreciated!
2013-01-17 10:58:50 utc MCamou thanks to you guys… this came up just as we were planning how to make our participant tests :)
2013-01-17 22:01:50 utc Skippy1 anyone have any experience subclassing a sequence expression?
2013-01-17 22:03:02 utc jmettraux Skippy1: hello, welcome to #ruote
2013-01-17 22:03:10 utc Skippy1 hey
2013-01-17 22:03:13 utc jmettraux I do have, but what is the goal?
2013-01-17 22:03:41 utc Skippy1 I'm trying to add some domain specific stuff to ruote
2013-01-17 22:04:20 utc jmettraux ok
2013-01-17 22:04:34 utc Skippy1 but when I run a test it hangs
2013-01-17 22:05:03 utc Skippy1 I managed to subclass iterator successfully
2013-01-17 22:05:20 utc jmettraux do you have a gist of the expression and of the test?
2013-01-17 22:05:32 utc jmettraux https://gist.github.com/
2013-01-17 22:05:38 utc Skippy1 just a sec
2013-01-17 22:08:12 utc Skippy1 here's the subclass
2013-01-17 22:08:14 utc Skippy1 https://gist.github.com/4560197
2013-01-17 22:08:31 utc Skippy1 and the test
2013-01-17 22:08:33 utc Skippy1 https://gist.github.com/4560183
2013-01-17 22:08:42 utc Skippy1 it echo's A
2013-01-17 22:08:44 utc Skippy1 but hangs
2013-01-17 22:08:46 utc Skippy1 no B
2013-01-17 22:09:25 utc jmettraux what does the NOISY=true output say?
2013-01-17 22:09:59 utc Skippy1 I set it in my environment file but I can't find any output in my logs....
2013-01-17 22:10:32 utc jmettraux NOISY=true ruby test/x/y/z/my_test.rb
2013-01-17 22:10:53 utc Skippy1 lemme try
2013-01-17 22:11:08 utc jmettraux unless you are using rspec...
2013-01-17 22:11:54 utc jmettraux Is the field "alert" set ? Does it contain an "infos" field ?
2013-01-17 22:12:18 utc jmettraux does the value held in the "infos" field respond to #first ?
2013-01-17 22:12:31 utc Skippy1 yep
2013-01-17 22:12:31 utc jmettraux and #any?
2013-01-17 22:12:37 utc Skippy1 it should be an array
2013-01-17 22:12:54 utc jmettraux should
2013-01-17 22:13:43 utc Skippy1 yeah it's an array... either empty or with hashes
2013-01-17 22:14:24 utc jmettraux I have the impression your line is breaking, noisy will tell
2013-01-17 22:14:25 utc Skippy1 hashs as elements
2013-01-17 22:15:33 utc Skippy1 ok think I've got noisy set up properly in my specs
2013-01-17 22:15:36 utc Skippy1 hold on
2013-01-17 22:15:50 utc jmettraux alternatively, if you don't have access to the STDOUT in your test environment, you could do process = ruote.ps(wfid); pp process.errors;
2013-01-17 22:16:26 utc Skippy1 https://gist.github.com/4560249
2013-01-17 22:16:31 utc Skippy1 that's what I get...
2013-01-17 22:16:52 utc Skippy1 on the screen
2013-01-17 22:17:05 utc Skippy1 nothing else in logs
2013-01-17 22:17:18 utc jmettraux noisy emits to STDOUT
2013-01-17 22:17:25 utc Skippy1 yeah
2013-01-17 22:17:29 utc Skippy1 from what I can tell
2013-01-17 22:17:42 utc Skippy1 it seems it thinks with_info is a storage participant?
2013-01-17 22:18:00 utc Skippy1 which would explain the hanging....
2013-01-17 22:18:32 utc jmettraux yes
2013-01-17 22:18:40 utc jmettraux well spotted :-)
2013-01-17 22:19:17 utc Skippy1 I do have names( :with_info ) in my expression definition...
2013-01-17 22:19:28 utc Skippy1 do I need anything more?
2013-01-17 22:19:47 utc jmettraux so you were successful with iterator subclassing?
2013-01-17 22:19:53 utc Skippy1 yeah
2013-01-17 22:20:27 utc Skippy1 https://gist.github.com/4560278
2013-01-17 22:20:31 utc Skippy1 works fine
2013-01-17 22:20:35 utc jmettraux ruote picks up new expressions at initialization, is the with_info code required before ruote gets initialized?
2013-01-17 22:21:39 utc Skippy1 it's required in the same file (my environment.rb) as the iterator
2013-01-17 22:22:20 utc Skippy1 ruote is required first... and then I require in my expressions
2013-01-17 22:22:51 utc jmettraux is the with_info required before ruote is initialized?
2013-01-17 22:23:44 utc Skippy1 looks like it in my spec helper
2013-01-17 22:24:08 utc jmettraux ok, try to put some debug output in https://github.com/jmettraux/ruote/blob/master/lib/ruote/svc/expression_map.rb#L66-L80
2013-01-17 22:24:18 utc jmettraux to see if your with_info expression gets picked
2013-01-17 22:24:48 utc Skippy1 ok
2013-01-17 22:26:23 utc Skippy1 just need to find that file in my rvm dir....
2013-01-17 22:26:39 utc jmettraux you can do: bundle open ruote
2013-01-17 22:27:56 utc Skippy1 ah
2013-01-17 22:28:02 utc Skippy1 found it anyway...
2013-01-17 22:29:51 utc Skippy1 I can see it adding other expressions...
2013-01-17 22:30:00 utc Skippy1 Added: ForEachInfoExpression -> Ruote::Exp::ForEachInfoExpression
2013-01-17 22:30:06 utc Skippy1 but no WithInfoExpression
2013-01-17 22:30:17 utc Skippy1 I know!
2013-01-17 22:30:24 utc Skippy1 It needs to be called WithInfoExpression
2013-01-17 22:30:28 utc Skippy1 it's called WithInfo
2013-01-17 22:30:31 utc Skippy1 d'oh
2013-01-17 22:30:34 utc jmettraux aaaah, well done!
2013-01-17 22:30:38 utc Skippy1 I see in the source it looks for that
2013-01-17 22:31:04 utc jmettraux great
2013-01-17 22:31:21 utc Skippy1 works!
2013-01-17 22:31:24 utc Skippy1 specs pass
2013-01-17 22:31:25 utc Skippy1 thx
2013-01-17 22:31:38 utc jmettraux you're welcome
2013-01-17 22:32:19 utc Skippy1 been bugging me the whole week
2013-01-17 22:32:31 utc Skippy1 anyway getting late
2013-01-17 22:32:40 utc jmettraux have a good night!
2013-01-17 23:30:33 utc ypz hi, john, howdy
2013-01-17 23:31:23 utc ypz I am playing with sequel storage, and I keep getting this error msg: ruote step error: #<Sequel::DatabaseDisconnectError: Mysql2::Error: MySQL server has gone away>
2013-01-17 23:31:42 utc ypz but the script still finishes successfully
2013-01-17 23:32:19 utc ypz sometimes, it writes out a stack trace, sometimes it doesn't
2013-01-17 23:36:25 utc jmettraux ypz: hello, this mysql2 is tricky, please try with the regular mysql driver
2013-01-17 23:36:47 utc ypz argh, good to know that
2013-01-17 23:37:15 utc ypz though I have been using mysql2 with my rails app without any issues
2013-01-17 23:38:09 utc jmettraux I've a CI server going on with mysql2 and it's always nagging me
2013-01-17 23:38:17 utc jmettraux Sequel::DatabaseDisconnectError: Mysql2::Error: This connection is still waiting for a result, try again once you have the result
2013-01-17 23:38:21 utc jmettraux not the same as your error
2013-01-17 23:38:27 utc jmettraux ok
2013-01-17 23:38:39 utc jmettraux maybe you can give me some more details about your test environment
2013-01-17 23:39:03 utc jmettraux and the complete error message
2013-01-17 23:40:51 utc jmettraux maybe with your help I can add some reconnect magic
2013-01-17 23:41:01 utc ypz this is on my macbook, mysql running locally, listening on unix socket; mysql2 gem is version 0.3.11
2013-01-17 23:41:36 utc ypz is there a good place to put my stack trace ? it's too much for irc,
2013-01-17 23:41:48 utc jmettraux https://gist.github.com/
2013-01-17 23:44:26 utc ypz https://gist.github.com/4560901
2013-01-17 23:47:34 utc ypz hm, how do you preserve spaces?
2013-01-17 23:48:47 utc jmettraux thanks, about spaces it should have been OK, .txt should have told him to <pre> it
2013-01-17 23:48:55 utc jmettraux it's OK, I can read it like that
2013-01-17 23:50:08 utc jmettraux I guess it works fine for Rails because it doesn't venture much in thread-land
2013-01-17 23:50:18 utc ypz IC
2013-01-17 23:50:47 utc jmettraux ok, let me look into the sequel mailing list about that one
2013-01-17 23:52:21 utc ypz I am testing with multiple workers (forked into sub processes) attached to the same sequel storage, when I do launch(pdef) , how do I get those workers to work on the process ?
2013-01-17 23:52:49 utc jmettraux ypz: sorry, let me focus on the first issue first
2013-01-17 23:52:57 utc ypz ok, thanks
2013-01-17 23:53:30 utc jmettraux well, it's easy though, if the worker are connected to the same storage, they'll compete for work
2013-01-17 23:53:47 utc jmettraux nothing to tell them, just hook them to the storage
2013-01-17 23:54:20 utc jmettraux although http://stackoverflow.com/questions/11640342/random-mysql-exceptions-in-my-rails-application
2013-01-17 23:54:30 utc jmettraux argh, not the same thing
2013-01-17 23:55:43 utc jmettraux there is something at https://github.com/brianmario/mysql2/issues/69
2013-01-17 23:58:03 utc jmettraux ypz: is the issue happening with your forked workers or can you see it with one worker (simplest setting) ?
2013-01-17 23:58:14 utc ypz that was my understanding how workers work, let me check if I forked it correctly
2013-01-17 23:58:27 utc jmettraux is the issue happening with your forked workers or can you see it with one worker (simplest setting) ?
2013-01-17 23:58:46 utc jmettraux is the issue happening with your forked workers or can you see it already with one worker (simplest setting) ?
2013-01-17 23:59:30 utc ypz with one worker , it works properly, I dumped out $$ and thread id, I can see that all work were done by multiple threads belonging to a single process
2013-01-17 23:59:41 utc jmettraux https://github.com/brianmario/mysql2/issues/69#issuecomment-4902557
2013-01-17 23:59:59 utc jmettraux I have the impression that what your are seeing is similar to ^^^
2013-01-18 00:00:00 utc ypz forked workers never went into action,
2013-01-18 00:00:34 utc jmettraux who emits the error message?
2013-01-18 00:00:44 utc jmettraux the main worker or a forked worker?
2013-01-18 00:01:31 utc ypz I can't tell, but would guess the main process
2013-01-18 00:01:58 utc jmettraux where do you get the output from?
2013-01-18 00:02:17 utc ypz yea, the main process, when doing engine.register
2013-01-18 00:02:28 utc jmettraux oko
2013-01-18 00:02:46 utc jmettraux may I see this uptime3.rb ?
2013-01-18 00:02:57 utc ypz sure, one sec
2013-01-18 00:05:54 utc ypz https://gist.github.com/4561016 well, the space is really messed up !!!
2013-01-18 00:06:55 utc jmettraux ok, please re-paste it, this time, please select "ruby" as the type of file, or enter something like "nada.rb" in the filename choice
2013-01-18 00:07:25 utc ypz let me work on that !
2013-01-18 00:11:20 utc jmettraux meanwhile, found interesting info: https://groups.google.com/forum/?fromgroups=#!topic/sequel-talk/wNoFul3dmEg