JPPF, java, parallel computing, distributed computing, grid computing, parallel, distributed, cluster, grid, cloud, open source, android, .net
JPPF

The open source
grid computing
solution

 Home   About   Features   Download   Documentation   On Github   Forums 
August 11, 2020, 09:39:35 PM *
Welcome,
Please login or register.

Login with username, password and session length
Advanced search  
News: New users, please read this message. Thank you!
  Home Help Search Login Register  
Pages: [1]   Go Down

Author Topic: Node connecting but not processing due to IP conflict  (Read 2119 times)

dbwiddis

  • JPPF Council Member
  • *****
  • Posts: 106
Node connecting but not processing due to IP conflict
« on: September 04, 2014, 09:15:58 PM »

Laurent,

We brought in a new member to our team and have most of our code working for him.  However, one thing that is NOT working is setting up a JPPF node on his machine.

Here's his node config file, with all the differences from defaults:
Quote
jppf.server.host = 192.168.100.59
jppf.server.port = 11111
jppf.discovery.enabled = false
jppf.jvm.options = -Xmx768m -Djava.util.logging.config.file=config/logging-node.properties
jppf.recovery.enabled = true
jppf.reconnect.initial.delay = 1
jppf.reconnect.max.time = -1
jppf.reconnect.interval = 5
jppf.socket.keepalive = true
jppf.processing.threads = 1

Starting the node seems to appear correctly on his local machine:
Quote
Attempting connection to the class server at 192.168.100.59:11111
RemoteClassLoaderConnection: Reconnected to the class server
JPPF Node management initialized on port 11198
Attempting connection to the node server at 192.168.100.59:11111
Reconnected to the node server
Node successfully initialized

However, the node never appears active on the admin GUI (Of note, server stats DOES show one connection in "number of nodes") and, more importantly, no tasks are ever sent to his node (indicating the driver is unaware of its existence/readiness to accept tasks.)

He is connected using Windows over a Cisco IPSEC VPN to the driver, and his IP address (on the VPN) is 192.168.237.14.  However, setting the logs to DEBUG shows what I think might be a problem; his JMX host is self-reporting his IP on his local network rather than the one for the VPN:
Quote
2014-09-04 13:59:05,972 [DEBUG][org.jppf.utils.NetworkUtils.getManagementHost(174)]: JMX host from NetworkUtils: 192.168.24.115
2014-09-04 13:59:05,972 [DEBUG][org.jppf.utils.NetworkUtils.getManagementHost(177)]: computed JMX host: 192.168.24.115
2014-09-04 13:59:05,972 [DEBUG][org.jppf.management.JMXMPServer.start(83)]: managementPort=11198, portProperties=[jppf.node.management.port, jppf.management.port]

We tried explicitly setting jppf.management.port in the node config file:
Quote
# JMX management host IP address. If not specified (recommended), the first non-local
# IP address (i.e. neither 127.0.0.1 nor localhost) on this machine will be used.
# If no non-local IP is found, localhost will be used.
jppf.management.host = 192.168.237.14

But while this change does appear in the logs, we still have the same symptoms (no GUI connection, no tasks sent).
Quote
2014-09-04 13:51:41,563 [DEBUG][org.jppf.utils.NetworkUtils.getManagementHost(174)]: JMX host from NetworkUtils: 192.168.24.115
2014-09-04 13:51:41,563 [DEBUG][org.jppf.utils.NetworkUtils.getManagementHost(177)]: computed JMX host: 192.168.237.14
2014-09-04 13:51:41,564 [DEBUG][org.jppf.management.JMXMPServer.start(83)]: managementPort=11198, portProperties=[jppf.node.management.port, jppf.management.port]

Of note, sometime during our debugging, we did end up seeing "phantom" node connections on the GUI (2 nodes showing the 192.168.237.14 IP, both on port 11198) but neither one responded to any management commands, leading me to suspect that some part of the connection/handshaking is occuring, but other parts are not.

Any other ideas what could be going wrong, or how to work around it?

Edit:  Additional symptoms.

While we had originally authorized the node program in Windows Firewall, we decided to disable the firewall completely and try again.  We got different symptoms.  The node did appear on the Admin GUI... and then again a few seconds later (same IP/port)... and then again... and so on until we shut it down with 6 duplicates on the list.   This was probably associated with the node recovery/reconnecting, but there's clearly still some issue with the initial handshake failing.
« Last Edit: September 04, 2014, 09:36:04 PM by dbwiddis »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Node connecting but not processing due to IP conflict
« Reply #1 on: September 05, 2014, 06:58:10 AM »

Hello,

Quote
However, the node never appears active on the admin GUI
Didn't you mean inactive?  Could you please confirm one way or the other?
[Edit: ok, scratch that, my eyes were definitely not fully opened  :)]
Could you also let us know where the GUI is running, is it on the same machine as the node or the driver, and is it connected to the driver via VPN?

I registered a bug for this issue: JPPF-320 Node reports unreachable JMX host in some configurations.
I'm currently investigating how to reproduce the issue your colleague is facing, by installing a VPN on my network (SoftEther). With a little bit of IP filtering on the JPPF side, I should be able to get sometihing similar. In any case, I will keep you apprised of my progress in this thread.

SIncerely,
-Laurent
« Last Edit: September 05, 2014, 08:19:27 AM by lolo »
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Node connecting but not processing due to IP conflict
« Reply #2 on: September 05, 2014, 10:05:08 AM »

I wanted to add that the JMX host you se in the node's log is in fact not used. The server will take whatever IP address the node uses to connect to the server port 11111. Remember that the admin UI is fowarding all node JMX requests through to the server, thus the IP address used by the server is the only one that matters.
To see what the server is getting, you can enable, in the server's log4j configuration, DEBUG level on the class "org.jppf.server.nio.nodeserver.AbstractNodeContext" and look in the log for messages like:
Code: [Select]
context RemoteNodeContext[...] setting management info [JPPFManagementInfo[host:port, type=node|MASTER, local=false, secure=false, uuid=some_uuid]]
-Laurent
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Node connecting but not processing due to IP conflict
« Reply #3 on: September 08, 2014, 06:33:45 PM »

Hello,

After quite a few tries, I managed to find one scenario which produced symptoms similar, but not identical, to the ones you are facing.
What I did:
- first, I installed a VPN software stack: SoftEther, an open-source L2TP/IPSec compatible VPN server and client. The VPN client was installed on 2 machines: driver machine and node machine
- I modified the code of the JMX connector server used by the driver and nodes, such that it now takes "jppf.management.host" into account. I also added a configuration property "jppf.management.host.wildcard = true|false" (true by default), as I realized, after re-reading the JMX remote specs, that the JMX connector server would by default bind to the wilcard address (i.e. bind to all network interfaces). With this property set to false, the jmx server will now only bind to the address, if any, specified by "jppf.management.host". You can get the new node jar from here.

In this setup, I have the following:
- the driver machine has the following IPs: 192.168.1.15 (LAN) and 169.254.237.246 (VPN)
- the node machine has the IPs: 192.168.1.21 (LAN) and 169.254.20.16 (VPN)

When in the node I set:
Code: [Select]
jppf.server.host = 169.254.237.246
jppf.management.host = 192.168.1.21
jppf.management.host.wildcard = false
Then I can see that the node connects to the driver via its 169.254.20.16 IP (VPN interface), but the driver is unable to connect ot the node's JMX server, because it is only bound to the node's LAN interface. As a consequence, in the admin console, the node does not show up in the topology view, however the driver statistcs do report that one node is connected.

Now, if I bind the node's JMX server to its VPN interface ("jppf.management.host = 169.254.20.16") instead of the LAN one, then all goes well, the node does show up in the admin console and is manageable.

So what I'm suspecting here is some kind of configuration issue in the firewall that prevents the driver from connecting to the node's JMX server.
Could you try and download the modified node jar, and configure the node with the following:
Code: [Select]
jppf.server.host = <driver_VPN_address>
jppf.management.host = <node_VPN_address>
jppf.management.host.wildcard = false
Can you also try and adjust the firewall settings to ensure that the management port is opened on the node's VPN interface, and let us know of the outcome?

Thanks,
-Laurent
Logged

dbwiddis

  • JPPF Council Member
  • *****
  • Posts: 106
Re: Node connecting but not processing due to IP conflict
« Reply #4 on: September 08, 2014, 07:24:18 PM »

Laurent,

Thanks for all your work on this issue.

We finally got it working by explicitly opening all 3 ports in the Windows firewall (11111, 11198, and 22222). 

I had initially thought we only needed 11198 and 22222 open, but for some reason we needed 11111 open for inbound as well.

(FYI to answer your questions we were running two copies of the GUI, one on the driver machine and one on my machine totally separate from everyone else's, and both got the same symptoms.)

I have not yet downloaded your updated jar.  Let me know if there's further testing we can/should do.

Dan
Logged

lolo

  • Administrator
  • JPPF Council Member
  • *****
  • Posts: 2262
    • JPPF Web site
Re: Node connecting but not processing due to IP conflict
« Reply #5 on: September 09, 2014, 07:10:27 PM »

Hi Dan,

Quote
We finally got it working by explicitly opening all 3 ports in the Windows firewall (11111, 11198, and 22222).
Great! Looks like the problem is resolved then.

Quote
Let me know if there's further testing we can/should do.
I don't think there's anything more to do. I will simply test further the interface binding feature for jMX and keep it in the distribution, since it's already there. Better to have it and not need it ...

Sincerely,
-Laurent
Logged
Pages: [1]   Go Up
 
JPPF Powered by SMF 2.0 RC5 | SMF © 2006–2011, Simple Machines LLC Get JPPF at SourceForge.net. Fast, secure and Free Open Source software downloads