Print

[codemonkey]

Hi Laurent, I observed that the jppf.local.execution.threads config property is no longer respected as of version 6.0

I have the following settings in my jppf-client.properties:

Code: [Select]

# Enable local execution of jobs? Default value is false (disabled)
jppf.local.execution.enabled = true

# Number of threads to use for local execution. Defaults to the number of CPUs available to the JVM
jppf.local.execution.threads = 16

On startup jppf reports that it will be running 8 processing threads, which is the available cores on my machine:

Code: [Select]

INFO  [2019-08-14T14:14:29,306] [main] {} JPPF Version: 6.0, Build number: 2240, Build date: 2018-10-06 09:32 CEST
INFO  [2019-08-14T14:14:29,307] [main] {} starting client with PID=21924, UUID=dc26d830-9c17-4553-a9e4-55fa6a39d800
INFO  [2019-08-14T14:14:29,307] [main] {} --------------------------------------------------------------------------------
INFO  [2019-08-14T14:14:50,936] [main] {} HikariCP libraries are not in the classpath, no datasource will be defined
INFO  [2019-08-14T14:14:55,704] [main] {} created client-side datasources: []
INFO  [2019-08-14T14:15:25,513] [main] {} load-balancer persistence is not configured
INFO  [2019-08-14T14:16:39,877] [main] {} running 8 processing threads
INFO  [2019-08-14T14:16:39,877] [main] {} Using default thread manager

Debugging the AbstractGenericClient class, the init method clearly just uses the availableProcessors:

Code: [Select]

protected void init(TypedProperties configuration)
{
/*.............*/
/* 139 */     int coreThreads = Runtime.getRuntime().availableProcessors();
/* 140 */     BlockingQueue<Runnable> queue = new SynchronousQueue();
/* 141 */     executor = new ThreadPoolExecutor(coreThreads, Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, queue, new JPPFThreadFactory("JPPF Client"));
/* 142 */     executor.allowCoreThreadTimeOut(true);
/* 143 */     if (jobManager == null) jobManager = createJobManager();
/* 144 */     Runnable r = new Runnable()
/*     */     {
/*     */       public void run() {
/* 147 */         initPools(config);
/*     */       }
/* 149 */     };
/* 150 */     ThreadUtils.startThread(r, "InitPools");
/*     */   }


Can you confirm? I need ability to use more than available cores to increase my throughput

Thank you in advance,

CM

[lolo]

Hi CM,

From my testing on v6.0.3, I do confirm that the "jppf.local.execution.threads" setting is honored by the JPPF client.
The code you showed from AbstractGenericClient.init() is creating a thread pool used for I/O and job scheduling, it is not creating the thread pool dedicated to tasks execution. The task execution pool is created in ClientExecutionManager whose constructor is invoked here.

So the problem must be elsewhere. Some possibilities:
- your client is not picking up the right configuration file
- "jppf.local.execution.threads" may be defined twice in your configuration, whith the second occurrence overriding the intended value

If none of these apply, could you post your client configuration, and any JPPFClient initialization code you deem relevant?

I also noticed, from your log extract, that you are running JPPF 6.0, would you consider upgrading to the latest maintenance release 6.0.3 ?

Thanks for your time,
-Laurent

[codemonkey]

Thank you Laurent for the quick reply on this.

I've confirmed that there is only one 'jppf.local.execution.threads' entry in the properties file as well confirmed that the value is set to 16, during debug when the file is loaded in the TypedProperties instance. I've attached the file for your reference.

Note: jppf.remote.execution.enabled doesn't exist in the properties file, I force that value to false in my code as I'm using local execution exclusively.

Here's the relevant code that initializes my JPPFClient:

Code: [Select]

private static JPPFClient jppfClient;
......

TypedProperties props = new TypedProperties();
try (InputStream is = new BufferedInputStream(new FileInputStream("path-to-props"))) {
props.load(is);
}
//force remote execution to false and local execution to true
props.setBoolean("jppf.remote.execution.enabled", false);
props.setBoolean("jppf.local.execution.enabled", true);

jppfClient = new JPPFClient(UUID.randomUUID().toString(), props, new ConnectionPoolListenerAdapter());


What could I be missing?

Thanks for the update on the maintenance release. I'm a little too close to the end of a release, so might not be feasible at this time

On anther note, I have a series of questions around local execution that I'll start a new thread on. 

Thank you for your time,

CM

[codemonkey]

Hi Laurent,

I've debugged this directly inside JPPFClient, outside of my application entirely. The method you pointed out in ChannelWrapperLocal never gets called.

Here we initialize the JobManager:
https://github.com/jppf-grid/JPPF/blob/7e44d321b8cbf07013cdec6a35ac35d53f7e8265/client/src/java/org/jppf/client/AbstractGenericClient.java#L157

Which initializes settings from the JPPFClient.config() in its constructor:

https://github.com/jppf-grid/JPPF/blob/7e44d321b8cbf07013cdec6a35ac35d53f7e8265/client/src/java/org/jppf/client/balancer/JobManagerClient.java#L127

At this time the JobManager knows the jppf client is local execution mode, as it has read it from the config of the JPPFClient

Then from AbstractGenericClient.initPools(),  setLocalExecutionEnabled() is called which then calls jobManager.setLocalExecutionEnabled(localExecutionEnabled):

https://github.com/jppf-grid/JPPF/blob/7e44d321b8cbf07013cdec6a35ac35d53f7e8265/client/src/java/org/jppf/client/AbstractGenericClient.java#L353


The following logic in jobManager.setLocalExecutionEnabled(localExecutionEnabled) evaluates to true, so updateLocalExecution() is never called, which creates ChannelWrapperLocal

Code: [Select]

    if (this.localEnabled == localExecutionEnabled) return;

https://github.com/jppf-grid/JPPF/blob/7e44d321b8cbf07013cdec6a35ac35d53f7e8265/client/src/java/org/jppf/client/balancer/JobManagerClient.java#L404

My test for reference:

Code: [Select]

    public static void main(String[] args) throws Exception {

        TypedProperties properties = new TypedProperties();
        try (InputStream is = new BufferedInputStream(new FileInputStream("/path/to/jppf-client.properties"))) {
            properties.load(is);
        }

        properties.setBoolean("jppf.remote.execution.enabled", false);
        properties.setBoolean("jppf.local.execution.enabled", true);
        log.info("jppf-client.properties: {}", properties);
        JPPFClient jc = new JPPFClient(UUID.randomUUID().toString(), properties, new ConnectionPoolListenerAdapter());

        log.info("JPPF Client local execution threads: {}", jc.getConfig().get(JPPFProperties.LOCAL_EXECUTION_THREADS));
        jc.close();
    }

Thank you advance for the assistance

CM

[lolo]

Hi CM,

Thank you very much for the detailed information, this was very helpful.
I finally managed to reproduce the behavior you described and found the cause of the issue. I registered a bug for this: JPPF-602 AbstractExecutionManager uses the wrong configuration to initialize

I will fix it in branches 6.0, 6.1 and 6.2 and prepare a patch for 6.0.0. This may take a couple of days, but it should be ready before next week. I will keep you apprised of the progress.

Sincerely,
-Laurent

[codemonkey]

This is great, thanks Laurent!

BTW, should community start using git to log issues? Makes total sense to use it vs the forums.

I've downloaded the jppf helm charts and see a few issues. Not sure if I should be posting them here, vs opening issues in git.

Thank you again!

CM

[lolo]

Hello CM,

Yes, you can definitely post issues on Github. I've been procrastinating on this because the current issue tracker was more convenient to me, and migrating to the Github issue system incurs some initial work. But in the end, what matters is what isconvenient ot the community. So, please go ahead!

-Laurent

[lolo]

Hi CM,

A fix for JPPF-602 was implemented and is now available in patch 01 for JPPF 6.0.

Please feel free to apply this patch at your convenience and let us know of the outcome.

Sincerely,
-Laurent

[codemonkey]

Hi Laurent,

Tests show that the processing threads I have configured are now being respected:

Code: [Select]

INFO  [2019-08-27T16:05:58,273] [main] {} JPPF Version: 6.0, Build number: 2242, Build date: 2019-08-24 10:46 CEST
INFO  [2019-08-27T16:05:58,273] [main] {} starting client with PID=11524, UUID=542564fd-b87f-4f5c-9b5e-c29c82c076e4
INFO  [2019-08-27T16:05:58,273] [main] {} --------------------------------------------------------------------------------
INFO  [2019-08-27T16:05:58,301] [main] {} HikariCP libraries are not in the classpath, no datasource will be defined
INFO  [2019-08-27T16:05:58,316] [main] {} created client-side datasources: []
INFO  [2019-08-27T16:05:58,363] [main] {} load-balancer persistence is not configured
INFO  [2019-08-27T16:05:58,426] [main] {} running 16 processing threads
INFO  [2019-08-27T16:05:58,426] [main] {} Using default thread manager

Thank you very much!

CM