ruote tmp/log_2013-03-22.html

2013-03-22 00:29:44 utc jmettraux blackdog: I gave the fix a second look, on_msg isn't conflicting with itself, it's probably check_waiting vs on_msg, so I unified the mutex: https://github.com/jmettraux/ruote/commit/6aacbbb08
2013-03-22 00:45:21 utc blackdog jmettraux: righto
2013-03-22 00:45:33 utc blackdog i'm still fighting nondeterminism in the tests :/
2013-03-22 00:45:40 utc blackdog (my app tests, i mean)
2013-03-22 01:05:06 utc jmettraux blackdog: still seeing the problem?
2013-03-22 01:06:32 utc jmettraux feel free to re-open the issue or to open a new one with the latest symptoms
2013-03-22 01:07:58 utc blackdog it takes forever to show up - i'll reopen if i see it
2013-03-22 01:08:04 utc blackdog trying to run now
2013-03-22 01:08:42 utc jmettraux curious to see if it's the same trace
2013-03-22 01:09:05 utc jmettraux going away from irc now, to work from a cafe
2013-03-22 01:09:39 utc jmettraux keeping an eye on the issue tracker, ttyl
2013-03-22 02:30:43 utc blackdog jmettraux: whee, upgrading ruote kills all my tests. might be a little while.
2013-03-22 03:01:20 utc jmettraux upgrading from which version?
2013-03-22 03:07:10 utc jmettraux may I help with the transition? Upgrading ruote shouldn't be painful
2013-03-22 03:30:19 utc blackdog 2.3.0.1
2013-03-22 03:31:45 utc blackdog it's just some failing tests - not as many as i'd thought
2013-03-22 03:31:54 utc blackdog we've got a very idiosyncratic way of handling errors
2013-03-22 03:32:08 utc blackdog ie, we catch them and hide them under the carpet :/
2013-03-22 03:56:03 utc jmettraux ok
2013-03-22 04:32:57 utc jmettraux heading out again
2013-03-22 08:01:26 utc jmettraux back
2013-03-22 09:47:09 utc richcorbs Hello, I cannot understand from the documentation if multiple tasks in a single job run sequentially or not. Anyone know?
2013-03-22 09:49:25 utc jmettraux richcorbs: hello, welcome to #ruote
2013-03-22 09:49:39 utc jmettraux please define "job" and "task"
2013-03-22 09:50:29 utc richcorbs job == "scheduler.cron '30 21 ***' do….end"
2013-03-22 09:50:44 utc richcorbs task == "do rake task #1"
2013-03-22 09:50:52 utc richcorbs task2 == "do rake task #2"
2013-03-22 09:51:25 utc jmettraux ah, so you're talking about rufus-scheduler
2013-03-22 09:52:28 utc richcorbs am I in the wrong place?
2013-03-22 09:52:50 utc jmettraux no, but your question could have been about ruote
2013-03-22 09:53:37 utc jmettraux ACTION browses the rufus-scheduler doc to give a decent answer
2013-03-22 09:54:07 utc jmettraux by default, each job gets its own thread of execution
2013-03-22 09:54:21 utc jmettraux so the execution will seem to overlap
2013-03-22 09:54:58 utc richcorbs ok, thanks
2013-03-22 09:55:05 utc jmettraux you're welcome
2013-03-22 09:55:34 utc ludovic_ hi John
2013-03-22 09:55:42 utc jmettraux ludovic_: hello
2013-03-22 09:55:48 utc jmettraux how are you doing?
2013-03-22 09:57:17 utc ludovic_ I've been thinking, we don't going to use amqp, I think we're going to implement ruote-resque. I don't feel like switching on rabbitmq just for ruote, it's already a huge migration for us
2013-03-22 09:57:53 utc jmettraux ludovic_: ah ok
2013-03-22 09:58:08 utc jmettraux for the record, I never suggested you to use ruote-amqp ;-)
2013-03-22 09:58:36 utc ludovic_ yeah yeah I know, it's just that the code is already up and running
2013-03-22 09:58:42 utc jmettraux and Resque wins over Sidekiq?
2013-03-22 09:59:38 utc ludovic_ Resque is maintained by github's guy, so the project has a good traction
2013-03-22 09:59:51 utc jmettraux ok
2013-03-22 10:00:12 utc ludovic_ The only upside of sidekiq is that it uses multiple threads
2013-03-22 10:00:35 utc ludovic_ If it's really safe, that should be on the roadmap of resque
2013-03-22 10:00:44 utc ludovic_ what do you think?
2013-03-22 10:01:31 utc richcorbs jmettraux: would scheduling multiple tasks within a single job with a mutex force the tasks to be run sequentially?
2013-03-22 10:02:11 utc jmettraux richcorbs: please read the readme at https://github.com/jmettraux/rufus-scheduler , it contains information about this subject
2013-03-22 10:03:16 utc jmettraux ludovic_: Resque has been around longer, the maintainer of Sidekiq is a smart guy, Sidekiq is based on Celluloid which is made by another smart guy, Resque has a big eco-system
2013-03-22 10:04:08 utc jmettraux Sidekiq has a growing eco-system, using multiple threads is not a bad thing if done correctly (especially when done correctly from the start)
2013-03-22 10:04:30 utc jmettraux it's a bit hard to decide
2013-03-22 10:04:54 utc jmettraux also, you guys' experience is important, are you a Resque or a Sidekiq shop?
2013-03-22 10:05:20 utc jmettraux you're probably already in production with one of those
2013-03-22 10:06:00 utc jmettraux about ruote, I don't know what'll you build, but it's already quite a jump, maybe it's safer to not jump with the queue system at the same time
2013-03-22 10:06:26 utc ludovic_ right now, we're embracing resque but we were thinking about a migration on sideqik
2013-03-22 10:09:03 utc ludovic_ roughly we ship an app that understand what you buy in grocery stores with a picture of your receipt
2013-03-22 10:10:03 utc ludovic_ so there's a lot of stuffs to do with our receipts (automated processing, outsourced processing, backoffice validations)
2013-03-22 10:10:44 utc jmettraux may I ask why bring ruote in the grocery back? ;-)
2013-03-22 10:10:50 utc jmettraux s/back/bag/
2013-03-22 10:11:03 utc ludovic_ sure :)
2013-03-22 10:12:25 utc richcorbs jmettraux: I see the example but it includes two separate jobs (different times) and not multiple tasks within a single job…guess I'll have to study up on what a mutex is.
2013-03-22 10:16:08 utc richcorbs If each task gets its own thread and I am constraining things to the specified mutex then it seems like it will do what I'm after.
2013-03-22 10:18:20 utc jmettraux if by task you mean a running instance of a job, then yes
2013-03-22 10:21:32 utc jmettraux I will add a note to the readme
2013-03-22 10:21:36 utc jmettraux thanks!
2013-03-22 10:22:23 utc ludovic_ actually, we ship an app but we propose also an sdk, so people can buy stuffs with our techno. Flows can be different according the source app (different features), and one flow is really complex. There's a lot of distributed actors (image processing, ocr, mechanical turk workers, back office validators) I want to able to know the flow and and receipt's state in it anytime.
2013-03-22 10:26:33 utc ludovic_ so people can build stuff ***
2013-03-22 10:26:39 utc jmettraux ok
2013-03-22 10:29:56 utc jmettraux makes sense
2013-03-22 10:31:19 utc richcorbs jmettraux: https://gist.github.com/richcorbs/5220305
2013-03-22 10:31:33 utc richcorbs both versions with/without mutex run the same locally for me is that what you'd expect?
2013-03-22 10:33:03 utc richcorbs both do what I'm trying to accomplish (run tasks sequentially within a single scheduled process)
2013-03-22 10:33:36 utc jmettraux so they don't overlap?
2013-03-22 10:33:50 utc richcorbs no but it could be my environment?
2013-03-22 10:34:18 utc richcorbs the numbers print with appropriate delays between them corresponding to the sleeps
2013-03-22 10:37:36 utc jmettraux could you add sample outputs to the gist?
2013-03-22 10:37:38 utc jmettraux please
2013-03-22 10:37:48 utc richcorbs sure
2013-03-22 10:39:41 utc richcorbs updated
2013-03-22 10:42:23 utc jmettraux ok, give me some time to react, in a meeting right now
2013-03-22 10:43:11 utc richcorbs no worries
2013-03-22 10:46:23 utc jmettraux so it never overlaps, the mutex is not used
2013-03-22 10:47:16 utc richcorbs I'm doing another test and will add it to the gist
2013-03-22 10:47:43 utc jmettraux so, so far it works as I want it to
2013-03-22 11:01:10 utc jmettraux so the job executes in 15s to 16s, it is scheduled every minute (at second 0), job instance at minute m+1 never overlaps job instance at minute m
2013-03-22 11:01:31 utc jmettraux with or without mutex it has the same output
2013-03-22 11:07:59 utc richcorbs sounds good
2013-03-22 11:08:02 utc richcorbs thanks for taking the time
2013-03-22 11:09:27 utc jmettraux you're welcome
2013-03-22 16:29:23 utc hoverlover Could someone please take a look at this simple code example and tell me if what I'm trying to do is possible? I'm basically trying to start a timer, allowing the rest of the process to immediately continue. Later, if the timer expires, I want to set a flag that basically tells the process to do something else. Here is a simple gist that shows what I'm trying to accomplish:
2013-03-22 16:29:25 utc hoverlover https://gist.github.com/hoverlover/92cd2c843cb9cec45eff
2013-03-22 21:28:59 utc jmettraux ok, added comment to https://gist.github.com/hoverlover/92cd2c843cb9cec45eff
2013-03-22 22:42:05 utc ypz jmettraux good morning
2013-03-22 22:46:02 utc ypz do you remember if any route users have encountered that "Virtual timer expired" error on systems running ruby 1.8.7 + ruote-sequel + mysql/mysql2 adapter ?
2013-03-22 22:46:18 utc ypz s/route/ruote/
2013-03-22 22:55:14 utc jmettraux ypz: hello, good evening
2013-03-22 22:55:24 utc jmettraux never seen that
2013-03-22 22:56:03 utc jmettraux http://stackoverflow.com/questions/3085968/rake-fails-with-virtual-timer-expired
2013-03-22 22:56:27 utc jmettraux http://www.ruby-forum.com/topic/209103
2013-03-22 22:56:42 utc jmettraux https://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&q=virtual+timer+expired
2013-03-22 22:58:02 utc jmettraux seems like people get rid of it by downgrading the mysql gem
2013-03-22 22:58:40 utc jmettraux I wonder what versions you use
2013-03-22 23:00:04 utc jmettraux you should also consider moving away from ruby 1.8, there won't be any security updates for it after this summer
2013-03-22 23:00:05 utc jmettraux iircv
2013-03-22 23:01:01 utc ypz i followed their instruction and used mysql 2.7, now it is partially working
2013-03-22 23:02:02 utc ypz I can talk to sequel storage to get the processes and their info, but the process does not proceed, stuck at the first participant
2013-03-22 23:02:40 utc ypz when I run strace against my worker process, I see that "Virtual timer expired" error,
2013-03-22 23:03:24 utc jmettraux ypz: you're not very clear
2013-03-22 23:03:40 utc jmettraux are all your processes using mysql 2.7 gem, what were they using before?
2013-03-22 23:04:16 utc jmettraux are you telling me that your worker process is using the mysql 2.7 gem and it's still emitting the "vt expired" error?
2013-03-22 23:04:26 utc ypz I used to use mysql2 adapter on *other* systems with ruby 1.9.3
2013-03-22 23:05:27 utc jmettraux ok, I can't help you, I cannot understand what you're running
2013-03-22 23:06:01 utc ypz I have been doing Ruote work in dev env until yesterday, now starting to move the app into production servers which only have 1.8.7 and start seeing this error
2013-03-22 23:06:50 utc jmettraux are all your processes in the production environment using the mysql gem 2.7, yes or no ?
2013-03-22 23:07:25 utc ypz there is only one production server which is using mysql 2.7
2013-03-22 23:08:31 utc ypz on this prod server, I still have this "virtual time expired" error
2013-03-22 23:08:48 utc jmettraux please show me the Gemfile for the worker process on the production server
2013-03-22 23:11:40 utc ypz https://gist.github.com/ypz/5225393
2013-03-22 23:12:15 utc jmettraux running 1.9.3 on dev and 1.8.7 on prod ?
2013-03-22 23:12:30 utc ypz yes, unfortunately
2013-03-22 23:13:35 utc jmettraux what version of the mysql gem are you using on dev?
2013-03-22 23:14:33 utc ypz I have been using mysql2 0.3.11 all this time in dev
2013-03-22 23:14:50 utc jmettraux why not use it on prod?
2013-03-22 23:15:22 utc ypz I see same "virtual time expired" error,
2013-03-22 23:15:26 utc ypz with msyql2
2013-03-22 23:15:55 utc ypz that' why I moved to mysql 2.7 after googling for the error and seeing other people's solutions
2013-03-22 23:16:31 utc jmettraux people downgraded from 2.8 to 2.7, but the latest is 2.9.1
2013-03-22 23:16:42 utc jmettraux did you give mysql gem 2.9.1 a try on prod?
2013-03-22 23:16:49 utc ypz no
2013-03-22 23:17:06 utc ypz i certainly can try it out
2013-03-22 23:17:08 utc jmettraux there is a chance the issue is gone after 2.8
2013-03-22 23:17:36 utc jmettraux also did you try msyql2 0.3.12 beta? It might contain a fix
2013-03-22 23:17:54 utc ypz no, not 0.3.12 beta
2013-03-22 23:17:58 utc jmettraux ACTION looks at https://rubygems.org/gems/mysql and https://rubygems.org/gems/mysql2 for version history
2013-03-22 23:18:32 utc jmettraux those google search seem to indicate it's a mysql[2] gem issue
2013-03-22 23:19:34 utc jmettraux you should request having a better ruby on the prod server, or downgrade your dev environment
2013-03-22 23:19:52 utc jmettraux ok, good luck
2013-03-22 23:19:56 utc ypz thanks
2013-03-22 23:37:38 utc jmettraux ypz: according to https://github.com/brianmario/mysql2/issues/51 the issue has no definite fix, the bug is closed "due to inactivity"
2013-03-22 23:37:47 utc jmettraux still appears on 0.3.12b4
2013-03-22 23:38:15 utc jmettraux the last comment is interesting, it puts "remote" in bold
2013-03-22 23:38:43 utc jmettraux is your mysql "remote" on production?
2013-03-22 23:38:50 utc ypz yes
2013-03-22 23:39:01 utc jmettraux and local in dev?
2013-03-22 23:40:16 utc jmettraux all the people in the bug thread are on Ruby 1.8
2013-03-22 23:40:44 utc ypz in dev I have db both local and remote in different dev setups ( I have more than one dev environments), but all dev env are using ruby 1.9.3
2013-03-22 23:41:00 utc jmettraux ok
2013-03-22 23:42:02 utc ypz I have been pushing our ops team for ruby upgrade, but ...
2013-03-22 23:42:20 utc jmettraux you could have asked for a staging environement too
2013-03-22 23:42:38 utc jmettraux make a smaller jump
2013-03-22 23:44:44 utc ypz well, there are other apps had to be considered as well, one of them is puppet, we are on puppet 2.7 which is not officially cleared for ruby 1.9.3
2013-03-22 23:45:04 utc jmettraux that means you cannot develop on 1.9.3
2013-03-22 23:46:45 utc jmettraux ok, breakfast now
2013-03-22 23:47:33 utc ypz thanks again
2013-03-22 23:47:37 utc jmettraux sorry for being grumpy, I just hate it when people ask for help and pass incomplete information
2013-03-22 23:47:47 utc jmettraux you're welcome