2013-03-22 00:29:44 utc |
jmettraux |
blackdog: I gave the fix a second look, on_msg isn't conflicting with itself, it's probably check_waiting vs on_msg, so I unified the mutex: https://github.com/jmettraux/ruote/commit/6aacbbb08 |
2013-03-22 00:45:21 utc |
blackdog |
jmettraux: righto |
2013-03-22 00:45:33 utc |
blackdog |
i'm still fighting nondeterminism in the tests :/ |
2013-03-22 00:45:40 utc |
blackdog |
(my app tests, i mean) |
2013-03-22 01:05:06 utc |
jmettraux |
blackdog: still seeing the problem? |
2013-03-22 01:06:32 utc |
jmettraux |
feel free to re-open the issue or to open a new one with the latest symptoms |
2013-03-22 01:07:58 utc |
blackdog |
it takes forever to show up - i'll reopen if i see it |
2013-03-22 01:08:04 utc |
blackdog |
trying to run now |
2013-03-22 01:08:42 utc |
jmettraux |
curious to see if it's the same trace |
2013-03-22 01:09:05 utc |
jmettraux |
going away from irc now, to work from a cafe |
2013-03-22 01:09:39 utc |
jmettraux |
keeping an eye on the issue tracker, ttyl |
2013-03-22 02:30:43 utc |
blackdog |
jmettraux: whee, upgrading ruote kills all my tests. might be a little while. |
2013-03-22 03:01:20 utc |
jmettraux |
upgrading from which version? |
2013-03-22 03:07:10 utc |
jmettraux |
may I help with the transition? Upgrading ruote shouldn't be painful |
2013-03-22 03:30:19 utc |
blackdog |
2.3.0.1 |
2013-03-22 03:31:45 utc |
blackdog |
it's just some failing tests - not as many as i'd thought |
2013-03-22 03:31:54 utc |
blackdog |
we've got a very idiosyncratic way of handling errors |
2013-03-22 03:32:08 utc |
blackdog |
ie, we catch them and hide them under the carpet :/ |
2013-03-22 03:56:03 utc |
jmettraux |
ok |
2013-03-22 04:32:57 utc |
jmettraux |
heading out again |
2013-03-22 08:01:26 utc |
jmettraux |
back |
2013-03-22 09:47:09 utc |
richcorbs |
Hello, I cannot understand from the documentation if multiple tasks in a single job run sequentially or not. Anyone know? |
2013-03-22 09:49:25 utc |
jmettraux |
richcorbs: hello, welcome to #ruote |
2013-03-22 09:49:39 utc |
jmettraux |
please define "job" and "task" |
2013-03-22 09:50:29 utc |
richcorbs |
job == "scheduler.cron '30 21 ***' do….end" |
2013-03-22 09:50:44 utc |
richcorbs |
task == "do rake task #1" |
2013-03-22 09:50:52 utc |
richcorbs |
task2 == "do rake task #2" |
2013-03-22 09:51:25 utc |
jmettraux |
ah, so you're talking about rufus-scheduler |
2013-03-22 09:52:28 utc |
richcorbs |
am I in the wrong place? |
2013-03-22 09:52:50 utc |
jmettraux |
no, but your question could have been about ruote |
2013-03-22 09:53:37 utc |
jmettraux |
ACTION browses the rufus-scheduler doc to give a decent answer |
2013-03-22 09:54:07 utc |
jmettraux |
by default, each job gets its own thread of execution |
2013-03-22 09:54:21 utc |
jmettraux |
so the execution will seem to overlap |
2013-03-22 09:54:58 utc |
richcorbs |
ok, thanks |
2013-03-22 09:55:05 utc |
jmettraux |
you're welcome |
2013-03-22 09:55:34 utc |
ludovic_ |
hi John |
2013-03-22 09:55:42 utc |
jmettraux |
ludovic_: hello |
2013-03-22 09:55:48 utc |
jmettraux |
how are you doing? |
2013-03-22 09:57:17 utc |
ludovic_ |
I've been thinking, we don't going to use amqp, I think we're going to implement ruote-resque. I don't feel like switching on rabbitmq just for ruote, it's already a huge migration for us |
2013-03-22 09:57:53 utc |
jmettraux |
ludovic_: ah ok |
2013-03-22 09:58:08 utc |
jmettraux |
for the record, I never suggested you to use ruote-amqp ;-) |
2013-03-22 09:58:36 utc |
ludovic_ |
yeah yeah I know, it's just that the code is already up and running |
2013-03-22 09:58:42 utc |
jmettraux |
and Resque wins over Sidekiq? |
2013-03-22 09:59:38 utc |
ludovic_ |
Resque is maintained by github's guy, so the project has a good traction |
2013-03-22 09:59:51 utc |
jmettraux |
ok |
2013-03-22 10:00:12 utc |
ludovic_ |
The only upside of sidekiq is that it uses multiple threads |
2013-03-22 10:00:35 utc |
ludovic_ |
If it's really safe, that should be on the roadmap of resque |
2013-03-22 10:00:44 utc |
ludovic_ |
what do you think? |
2013-03-22 10:01:31 utc |
richcorbs |
jmettraux: would scheduling multiple tasks within a single job with a mutex force the tasks to be run sequentially? |
2013-03-22 10:02:11 utc |
jmettraux |
richcorbs: please read the readme at https://github.com/jmettraux/rufus-scheduler , it contains information about this subject |
2013-03-22 10:03:16 utc |
jmettraux |
ludovic_: Resque has been around longer, the maintainer of Sidekiq is a smart guy, Sidekiq is based on Celluloid which is made by another smart guy, Resque has a big eco-system |
2013-03-22 10:04:08 utc |
jmettraux |
Sidekiq has a growing eco-system, using multiple threads is not a bad thing if done correctly (especially when done correctly from the start) |
2013-03-22 10:04:30 utc |
jmettraux |
it's a bit hard to decide |
2013-03-22 10:04:54 utc |
jmettraux |
also, you guys' experience is important, are you a Resque or a Sidekiq shop? |
2013-03-22 10:05:20 utc |
jmettraux |
you're probably already in production with one of those |
2013-03-22 10:06:00 utc |
jmettraux |
about ruote, I don't know what'll you build, but it's already quite a jump, maybe it's safer to not jump with the queue system at the same time |
2013-03-22 10:06:26 utc |
ludovic_ |
right now, we're embracing resque but we were thinking about a migration on sideqik |
2013-03-22 10:09:03 utc |
ludovic_ |
roughly we ship an app that understand what you buy in grocery stores with a picture of your receipt |
2013-03-22 10:10:03 utc |
ludovic_ |
so there's a lot of stuffs to do with our receipts (automated processing, outsourced processing, backoffice validations) |
2013-03-22 10:10:44 utc |
jmettraux |
may I ask why bring ruote in the grocery back? ;-) |
2013-03-22 10:10:50 utc |
jmettraux |
s/back/bag/ |
2013-03-22 10:11:03 utc |
ludovic_ |
sure :) |
2013-03-22 10:12:25 utc |
richcorbs |
jmettraux: I see the example but it includes two separate jobs (different times) and not multiple tasks within a single job…guess I'll have to study up on what a mutex is. |
2013-03-22 10:16:08 utc |
richcorbs |
If each task gets its own thread and I am constraining things to the specified mutex then it seems like it will do what I'm after. |
2013-03-22 10:18:20 utc |
jmettraux |
if by task you mean a running instance of a job, then yes |
2013-03-22 10:21:32 utc |
jmettraux |
I will add a note to the readme |
2013-03-22 10:21:36 utc |
jmettraux |
thanks! |
2013-03-22 10:22:23 utc |
ludovic_ |
actually, we ship an app but we propose also an sdk, so people can buy stuffs with our techno. Flows can be different according the source app (different features), and one flow is really complex. There's a lot of distributed actors (image processing, ocr, mechanical turk workers, back office validators) I want to able to know the flow and and receipt's state in it anytime. |
2013-03-22 10:26:33 utc |
ludovic_ |
so people can build stuff *** |
2013-03-22 10:26:39 utc |
jmettraux |
ok |
2013-03-22 10:29:56 utc |
jmettraux |
makes sense |
2013-03-22 10:31:19 utc |
richcorbs |
jmettraux: https://gist.github.com/richcorbs/5220305 |
2013-03-22 10:31:33 utc |
richcorbs |
both versions with/without mutex run the same locally for me is that what you'd expect? |
2013-03-22 10:33:03 utc |
richcorbs |
both do what I'm trying to accomplish (run tasks sequentially within a single scheduled process) |
2013-03-22 10:33:36 utc |
jmettraux |
so they don't overlap? |
2013-03-22 10:33:50 utc |
richcorbs |
no but it could be my environment? |
2013-03-22 10:34:18 utc |
richcorbs |
the numbers print with appropriate delays between them corresponding to the sleeps |
2013-03-22 10:37:36 utc |
jmettraux |
could you add sample outputs to the gist? |
2013-03-22 10:37:38 utc |
jmettraux |
please |
2013-03-22 10:37:48 utc |
richcorbs |
sure |
2013-03-22 10:39:41 utc |
richcorbs |
updated |
2013-03-22 10:42:23 utc |
jmettraux |
ok, give me some time to react, in a meeting right now |
2013-03-22 10:43:11 utc |
richcorbs |
no worries |
2013-03-22 10:46:23 utc |
jmettraux |
so it never overlaps, the mutex is not used |
2013-03-22 10:47:16 utc |
richcorbs |
I'm doing another test and will add it to the gist |
2013-03-22 10:47:43 utc |
jmettraux |
so, so far it works as I want it to |
2013-03-22 11:01:10 utc |
jmettraux |
so the job executes in 15s to 16s, it is scheduled every minute (at second 0), job instance at minute m+1 never overlaps job instance at minute m |
2013-03-22 11:01:31 utc |
jmettraux |
with or without mutex it has the same output |
2013-03-22 11:07:59 utc |
richcorbs |
sounds good |
2013-03-22 11:08:02 utc |
richcorbs |
thanks for taking the time |
2013-03-22 11:09:27 utc |
jmettraux |
you're welcome |
2013-03-22 16:29:23 utc |
hoverlover |
Could someone please take a look at this simple code example and tell me if what I'm trying to do is possible? I'm basically trying to start a timer, allowing the rest of the process to immediately continue. Later, if the timer expires, I want to set a flag that basically tells the process to do something else. Here is a simple gist that shows what I'm trying to accomplish: |
2013-03-22 16:29:25 utc |
hoverlover |
https://gist.github.com/hoverlover/92cd2c843cb9cec45eff |
2013-03-22 21:28:59 utc |
jmettraux |
ok, added comment to https://gist.github.com/hoverlover/92cd2c843cb9cec45eff |
2013-03-22 22:42:05 utc |
ypz |
jmettraux good morning |
2013-03-22 22:46:02 utc |
ypz |
do you remember if any route users have encountered that "Virtual timer expired" error on systems running ruby 1.8.7 + ruote-sequel + mysql/mysql2 adapter ? |
2013-03-22 22:46:18 utc |
ypz |
s/route/ruote/ |
2013-03-22 22:55:14 utc |
jmettraux |
ypz: hello, good evening |
2013-03-22 22:55:24 utc |
jmettraux |
never seen that |
2013-03-22 22:56:03 utc |
jmettraux |
http://stackoverflow.com/questions/3085968/rake-fails-with-virtual-timer-expired |
2013-03-22 22:56:27 utc |
jmettraux |
http://www.ruby-forum.com/topic/209103 |
2013-03-22 22:56:42 utc |
jmettraux |
https://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&q=virtual+timer+expired |
2013-03-22 22:58:02 utc |
jmettraux |
seems like people get rid of it by downgrading the mysql gem |
2013-03-22 22:58:40 utc |
jmettraux |
I wonder what versions you use |
2013-03-22 23:00:04 utc |
jmettraux |
you should also consider moving away from ruby 1.8, there won't be any security updates for it after this summer |
2013-03-22 23:00:05 utc |
jmettraux |
iircv |
2013-03-22 23:01:01 utc |
ypz |
i followed their instruction and used mysql 2.7, now it is partially working |
2013-03-22 23:02:02 utc |
ypz |
I can talk to sequel storage to get the processes and their info, but the process does not proceed, stuck at the first participant |
2013-03-22 23:02:40 utc |
ypz |
when I run strace against my worker process, I see that "Virtual timer expired" error, |
2013-03-22 23:03:24 utc |
jmettraux |
ypz: you're not very clear |
2013-03-22 23:03:40 utc |
jmettraux |
are all your processes using mysql 2.7 gem, what were they using before? |
2013-03-22 23:04:16 utc |
jmettraux |
are you telling me that your worker process is using the mysql 2.7 gem and it's still emitting the "vt expired" error? |
2013-03-22 23:04:26 utc |
ypz |
I used to use mysql2 adapter on *other* systems with ruby 1.9.3 |
2013-03-22 23:05:27 utc |
jmettraux |
ok, I can't help you, I cannot understand what you're running |
2013-03-22 23:06:01 utc |
ypz |
I have been doing Ruote work in dev env until yesterday, now starting to move the app into production servers which only have 1.8.7 and start seeing this error |
2013-03-22 23:06:50 utc |
jmettraux |
are all your processes in the production environment using the mysql gem 2.7, yes or no ? |
2013-03-22 23:07:25 utc |
ypz |
there is only one production server which is using mysql 2.7 |
2013-03-22 23:08:31 utc |
ypz |
on this prod server, I still have this "virtual time expired" error |
2013-03-22 23:08:48 utc |
jmettraux |
please show me the Gemfile for the worker process on the production server |
2013-03-22 23:11:40 utc |
ypz |
https://gist.github.com/ypz/5225393 |
2013-03-22 23:12:15 utc |
jmettraux |
running 1.9.3 on dev and 1.8.7 on prod ? |
2013-03-22 23:12:30 utc |
ypz |
yes, unfortunately |
2013-03-22 23:13:35 utc |
jmettraux |
what version of the mysql gem are you using on dev? |
2013-03-22 23:14:33 utc |
ypz |
I have been using mysql2 0.3.11 all this time in dev |
2013-03-22 23:14:50 utc |
jmettraux |
why not use it on prod? |
2013-03-22 23:15:22 utc |
ypz |
I see same "virtual time expired" error, |
2013-03-22 23:15:26 utc |
ypz |
with msyql2 |
2013-03-22 23:15:55 utc |
ypz |
that' why I moved to mysql 2.7 after googling for the error and seeing other people's solutions |
2013-03-22 23:16:31 utc |
jmettraux |
people downgraded from 2.8 to 2.7, but the latest is 2.9.1 |
2013-03-22 23:16:42 utc |
jmettraux |
did you give mysql gem 2.9.1 a try on prod? |
2013-03-22 23:16:49 utc |
ypz |
no |
2013-03-22 23:17:06 utc |
ypz |
i certainly can try it out |
2013-03-22 23:17:08 utc |
jmettraux |
there is a chance the issue is gone after 2.8 |
2013-03-22 23:17:36 utc |
jmettraux |
also did you try msyql2 0.3.12 beta? It might contain a fix |
2013-03-22 23:17:54 utc |
ypz |
no, not 0.3.12 beta |
2013-03-22 23:17:58 utc |
jmettraux |
ACTION looks at https://rubygems.org/gems/mysql and https://rubygems.org/gems/mysql2 for version history |
2013-03-22 23:18:32 utc |
jmettraux |
those google search seem to indicate it's a mysql[2] gem issue |
2013-03-22 23:19:34 utc |
jmettraux |
you should request having a better ruby on the prod server, or downgrade your dev environment |
2013-03-22 23:19:52 utc |
jmettraux |
ok, good luck |
2013-03-22 23:19:56 utc |
ypz |
thanks |
2013-03-22 23:37:38 utc |
jmettraux |
ypz: according to https://github.com/brianmario/mysql2/issues/51 the issue has no definite fix, the bug is closed "due to inactivity" |
2013-03-22 23:37:47 utc |
jmettraux |
still appears on 0.3.12b4 |
2013-03-22 23:38:15 utc |
jmettraux |
the last comment is interesting, it puts "remote" in bold |
2013-03-22 23:38:43 utc |
jmettraux |
is your mysql "remote" on production? |
2013-03-22 23:38:50 utc |
ypz |
yes |
2013-03-22 23:39:01 utc |
jmettraux |
and local in dev? |
2013-03-22 23:40:16 utc |
jmettraux |
all the people in the bug thread are on Ruby 1.8 |
2013-03-22 23:40:44 utc |
ypz |
in dev I have db both local and remote in different dev setups ( I have more than one dev environments), but all dev env are using ruby 1.9.3 |
2013-03-22 23:41:00 utc |
jmettraux |
ok |
2013-03-22 23:42:02 utc |
ypz |
I have been pushing our ops team for ruby upgrade, but ... |
2013-03-22 23:42:20 utc |
jmettraux |
you could have asked for a staging environement too |
2013-03-22 23:42:38 utc |
jmettraux |
make a smaller jump |
2013-03-22 23:44:44 utc |
ypz |
well, there are other apps had to be considered as well, one of them is puppet, we are on puppet 2.7 which is not officially cleared for ruby 1.9.3 |
2013-03-22 23:45:04 utc |
jmettraux |
that means you cannot develop on 1.9.3 |
2013-03-22 23:46:45 utc |
jmettraux |
ok, breakfast now |
2013-03-22 23:47:33 utc |
ypz |
thanks again |
2013-03-22 23:47:37 utc |
jmettraux |
sorry for being grumpy, I just hate it when people ask for help and pass incomplete information |
2013-03-22 23:47:47 utc |
jmettraux |
you're welcome |