Comments on: RetriableTask — a generic wrapper for retrying operations in Java

By: Andrew

Andrew — Sun, 12 Jul 2009 13:24:11 +0000

Yep, I can see this might require a slightly different approach if you’re doing webcrawling. In our case, we’re using this to dispatch parallel SOAP messages to various different web services living on different servers, so even before I brought this class in, we were still wrapping all the SOAP client operations in Callables and running them on a thread pool.

As Bob says, it would get more involved if you don’t want multiple threads contacting the same server at once. Although a quick way to adapt this class, for simple cases, would be to use a separate single-threaded executor for each server.

I love the concurrency framework.

By: Bob Carpenter

Bob Carpenter — Fri, 10 Jul 2009 20:34:36 +0000

For web downloads, you often have a whole set of URLs to download. It’s bad form to run multiple concurrent downloads against the same site/server.

So the usual approach to doing this runs a multi-threaded scheduling priority queue that orders things to download by time-to-next-try and doesn’t allow concurrent downloads from the same site (basically a set-based semaphore setup). Typically, under errors, there will be some scheme to wait longer before the next try. And often scheduling that depends on the last time you found fresh content.

I think rzei’s right, that at least a more generic abstract base class (not a whole framework!) might be useful here. A one-class alternative would be to take a set of exception Class objects, with exceptions in the set not being retried.

By: A slightly cleaner Java Retryable - mccv

A slightly cleaner Java Retryable - mccv — Thu, 28 May 2009 00:30:44 +0000

[...] implementation of a class that retries an operation N times before failing. However after reading this post I realized that using Java’s Callable class cleans things up quite a bit. This completely [...]

By: Andrew

Andrew — Tue, 26 May 2009 09:43:55 +0000

@mccv — It did occur to me as I was writing it that it would be much easier in a language where you could pass function calls around…

Callable is well worth getting to know though, as are ExecutorService, Future and FutureTask. Even if you don’t do industrial-grade concurrency, they give you some handy options for breaking your code into units of work.

By: mccv

mccv — Mon, 25 May 2009 17:36:17 +0000

Funny, I had a similar requirement and came up with a similar approach. My lack of familiarity with Callable shows, as I basically created the same interface. I put up a post comparing the implementation in Java vs. an implementation in Scala here http://www.themcwongs.com/?p=50

By: Andrew

Andrew — Sun, 24 May 2009 18:09:57 +0000

@Jkilgrow — heh, web services are exactly why I wrote this

By: Jkilgrow

Jkilgrow — Sun, 24 May 2009 14:26:11 +0000

I wrote something very similar for calling webservices that were particularly unreliable.

By: Andrew

Andrew — Sun, 24 May 2009 09:27:54 +0000

@Marcio — exactly, that’s what I intended. You get parallelism and timeouts for free that way. I should put mention that in the post really.

@rzei — sounds like that’s starting to grow into a proper framework rather than a one class solution I’m not sure why you think this method lacks reusability though.

By: Marcio

Marcio — Sat, 23 May 2009 17:10:00 +0000

@florin

Because it’s a Callable nothing stops you from executing it in an ExecutorService.

By: florin

florin — Sat, 23 May 2009 15:38:53 +0000

Wouldn’t you want this in its own thread, asynchronously especially if you put a timer on it? Also, a callback might be needed rather than a return on the call() method.