First initialization of my application takes more that 2 minutes (not optimized, need ~10 JARs - tons of classes + resource files)
It can improve startup time on slow networks with huge latencies. And decrease communication traffic between JPPF components.
JARs can be stored in temp directory and deleted on JPPFNode shutdown. Now class files and resources are cached in JPPFClassLoader by JPPFClient.ID.
I suggest extends JPPFClient and JPPFJob with new methods.
JPPFClient:addClassLoaderURL(URL)
JPPFJob:addClassLoaderURL(URL)
addClassLoaderURL - computes unique hashCode for URL (eg SHA-256 from URL content)
On creation of JPPFClassLocader the JPPFNode knows what unique hashCodes (ClassLoaderURLs) are need to download -> downloads URLs content from client
JPPFClassLocader extends URLClassLoader that has protected method addURL(URL url) but JPPFClassLocader uses only internal cache for class and resource search.
I have made simple JPPFTaskDownloader(final File... files) that downloads files to JPPFNode. It took only ~25 seconds to download several megabytes on slow WiFi, not 2 minutes.
public class JPPFTaskDownloader {
private final List<File> source = new ArrayList<File>();
public JPPFTaskDownloader(final Collection<File> files) {
source.addAll(files);
}
public void run() {
System.out.println("Task init");
long totalDur = System.nanoTime();
try {
ClientDataProvider dataProvider = (ClientDataProvider) getDataProvider();
ClassLoader classLoader = getClass().getClassLoader();
if(classLoader instanceof AbstractJPPFClassLoader && !source.isEmpty()) {
JPPFClassLoader jppfClassLoader = (JPPFClassLoader) classLoader;
for (File file : source) {
long dur = System.nanoTime();
byte[] data = safeComputeValue(dataProvider, file.getAbsolutePath(), new DownloadFile(file));
File out = File.createTempFile("jppf-", "-download");
FileOutputStream os = null;
try {
os = new FileOutputStream(out);
os.write(data);
} finally {
if(os != null) os.close();
}
dur = System.nanoTime() - dur;
System.out.printf("Adding file: %s - source: %s - size: %d, dur: %f%n", out.getName(), file.getName(), data.length, dur / 1000000.0);
file.deleteOnExit();
jppfClassLoader.addURL(file.toURI().toURL());
}
}
} catch (Exception e) {
e.printStackTrace();
setException(e);
} catch (Throwable t) {
t.printStackTrace();
setException(new JPPFException(t));
} finally {
totalDur = System.nanoTime() - totalDur;
System.out.println("FINSIHED in: " + (totalDur / 1000000.0));
}
}
public static class DownloadFile implements JPPFCallable<byte[]> {
private static final long serialVersionUID = 8363386268214414851L;
private final File file;
public DownloadFile(final File file) {
this.file = file;
}
@Override
public byte[] call() throws Exception {
System.out.printf("Downloading: %s%n", file.getAbsolutePath());
FileInputStream is = null;
byte[] result;
try {
is = new FileInputStream(file);
result = new byte[(int) file.length()];
readFully(is, result);
} finally {
if(is != null) is.close();
}
return result;
}
protected static void readFully(final FileInputStream in, final byte b[]) throws IOException {
int n = 0;
while (n < b.length) {
int count = in.read(b, n, b.length - n);
if (count < 0) throw new EOFException();
n += count;
}
}
}
public static <V> V safeComputeValue(final ClientDataProvider dataProvider, final Object key, final JPPFCallable<V> callable) {
if (key == null) return null;
Object result = dataProvider.computeValue(key, callable);
if (result instanceof Throwable) {
// System.out.println("Result: " + result);
((Throwable) result).printStackTrace();
}
return (V) result;
}
}
SHA-256
MessageDigest digest = MessageDigest.getInstance("SHA-256");
digest.reset();
ByteArrayOutputStream bos = new ByteArrayOutputStream();
...
bos.write(<data>)
...
bos.close();
byte[] hashCode = digest.digest(bos.toByteArray());
Cached file
File file = File.createTemporaryFile("jppf-", "-" + toHex(hashCode))...
file.deleteOnExit();
...
