Print

[brian]

Hi,

What would be the best place to hook in to add authentication and authorization to all socket connections which are opened in jppf? I'd like to ensure that any node, driver, client, visualvm instance, management gui, etc is running as user in appropriate role, using for example spring security before allowing them any read or write access. For me this would require a java.net.Socket instance on both sides to mutually handshake user names and check entitlements. SSL/TLS won't work for my case.

Looking at the code it seems JPPFDataTransform comes closest to the hook I need but
- I only need an up-front handshake, all other bytes can transport as is, so the overhead is unnecessary
- I'm not sure if it gets used for all connections (eg JMX)

If this is not possible today, I'd be happy to take a stab adding such a hook if you could give a pointer for how you'd like it done.

Thanks.

[lolo]

Hello,

I apologize for the late answer, your request got me thinking a lot

As of today, there is nothing in the JPPF design that allows for a pluggable and reliable authentication/authorization mechanism. What is available is certificate-based authentication when the communications are secured with SSL/TLS. However, as is your case, this is not always workable or convenient.

As you mentioned, the JPPFDataTransform has the disadvantages of being applied to all network traffic rather than just handshaking, and it also doesn't work with our JMX-based APIs.

I've been considering this feature for a long time now, and I even registered a feature request for this: JPPF-19 Enable access control. It was originally targeted at JPPF 4.0, but I kept pushing it back ever since. I guess that makes me a world-class procrastinator .

Some architecture and design considerations:

- we need a pluggable mechanism. Typically in JPPF this is done either via SPI or configuration properties. We even have a unified hook API to handle these use cases. My preference would go to SPI, because it doesn't expose any part of the hook via the configuration and is thus more secure

- the authn/authz mechanism must be very flexible and allow any type of authentication mechanism. Relying on Java's JAAS seems a workable approach, even if a rather complex one, but that's the usual tradeoff for flexibility ...

- all network connection handshaking instances must be considered: node to driver, client to driver, driver to driver and JMX remote connections. There is also the heartbeat-based recovery mechanism to deal with.

- it must also work in non-pure Java environments, i.e. .Net bridge and Android nodes.

- implementing authorizations will require a signifcant redesign, especially of the JMX-based management and monitoring APIs. The hook API will need to provide facilities for mapping custom mechansms to the JPPF authz model - which also needs to be specified and implemented.

All this to say that I'll welcome any help and contribution, however this is a very complex and long-term work. This is why it should be considered for a new major version (v6.0 in this instance), because it will most likely break compatibility with prior versions.

Sincerely,
-Laurent

[brian]

Thank you for the thoughtful reply.

I wonder if as an alternate route, you would consider a "phase 1" commit which avoids all notion of authn/authz and simply gives user opportunity to run some code immediately after socket establish? Similar to hazelcast's SocketInterceptor.[1] This would enable users like me to add coarse grained entitlements in a minor release!

[1] http://docs.hazelcast.org/docs/3.5/manual/html/socketinterceptor.html

[lolo]

Hello,

Sure, we could consider such a hook for the upcoming v5.2. I would just like to understand what scope it should have, so if that's ok with you, let's brainstorm a little.

- The first interrogation I have, is what kind of connections should we looking at?
Remember, a "logical" connection from node to driver, client to driver or driver to driver, actually encompasses 2 network connections: one for the distributed class loader, the other for the jobs. My take on this would be to not bother and invoke the hook for every actual network connection. Would you agree with this?

- What to do with the JMX connections? 
I looked at the Hazelcast doc for which you provided a link and, as far as I can tell, they do not have any socket interceptor hook for their JMX connections. The problem here is that, if you intend to use the hook for authentication purposes, then the JMX connections can open a big security hole.

Regarding the JMXMP connector used by JPPF, I believe it is possible to pass the hook as an environment parameter in the env map passed to JMXConnectorServerFactory.newJMXConnectorServer() and  JMXConnectorFactory.connect(), so it sounds doable. The implication is that the jxmremote code would then have a dependency on JPPF code at compile time, whereas currently JPPF has a runtime-only dependency on jmxremote. Do you have any suggestions on this? One possible option would be to use reflection to invoke the hook ... performance wouldn't be an issue, since the invocation would occur only once per connection.

- What data should be exposed in the hook? After looking at the Hazelcast interceptor feature (thanks for the link), I see that they expose raw sockets. I'm reluctant to expose raw sockets like that, because I think there's a lot of potential for mischief. I would rather encapsulate the information in a specific object type (e. g. SocketInfo with getHost() and getPort() methods). What's your take on this?

- in addition to the implementation, testing and documentation work, I think it would be nice to propose a dedicated sample in our samples pack. Any idea?

- finally, this feature will likely push the 5.2 release by a couple of weeks. However, I'm already so much behind with regards to my original timeline that I'm not sure it will make a huge difference. In any case, I will release another beta very soon, as a follow up to the testing and bug fixing I've done these last weeks.

Thanks,
-Laurent

[brian]

Thanks again for the thoughtful reply. I'm glad you would consider such a hook in 5.2.

- The first interrogation I have, is what kind of connections should we looking at?

Yes I agree, invoking hook for every network connection sounds simple and good. May be nice to also have the invocation tell the hook about the type of connection being established, eg {localComponent = driver, remoteComponent = node, connectionPurpose = classLoader} to support flexible authorization impls.

- What to do with the JMX connections?

I'd be using a different protocol than jmxmp, a custom one which includes auth. I add the same protocol to visualvm and other clients' classpath so they can talk. So I'd like to avoid some of the assumptions made in org.jppf.management.JMXMPServer. Perhaps a good approach here would be making org.jppf.management.JMXServerFactory pluggable along with its client counterpart(s) so JMX connection protocol is not hard-coded.

- What data should be exposed in the hook? After looking at the Hazelcast interceptor feature (thanks for the link), I see that they expose raw sockets. I'm reluctant to expose raw sockets like that, because I think there's a lot of potential for mischief.

Our authentication handshake does some asymmetric back and forth on the socket, hmm not sure how to do that without the Socket. Maybe if you were more specific about the type of mischief concerning you we could think of ways to deter it?

- in addition to the implementation, testing and documentation work, I think it would be nice to propose a dedicated sample in our samples pack. Any idea?

Perhaps a simple example where all components (node,driver,client,visualvm) only allow connection when userid of local process matches userid of remote process. Could use user.name system property to fetch userid, with comment saying it's not actually a secure impl.

[brian]

By the way, looks like an open source example of custom JMX protocol is here https://github.com/jbossas/remoting-jmx with an article about it here https://dzone.com/articles/remote-jmx-access-wildfly-or

[lolo]

Hello,

Thanks a lot for your feedback.

First I must say that I realized I completely missed the point of the interceptor, which is to provide a way to hijack the network connection for whatever purpose, including authn/authz. Now I got it. Therefore, my concern about providing raw sockets is totally pointless and I'd be grateful if we could just discard it And I apologize for the time wasted.

After this realization, I felt the need to see concretely how it could work, so I implemented a mechanism based on an interface named NetworkConnectionInterceptor, defined like this:

Code: [Select]

public interface NetworkConnectionInterceptor {
  // Called when a {@link Socket} is accepted by a {@link ServerSocket}
  boolean onAccept(Socket acceptedSocket);

  // Called when a {@link SocketChannel} is accepted by a {@link ServerSocketChannel}
  boolean onAccept(SocketChannel acceptedSocket);

  // Called when a {@link Socket} is connected
  boolean onConnect(Socket connectedSocket);
}
Note the onAccept() method that takes a SocketChannel parameter. The driver's network communications are based on non-blocking I/O (as specified in java.nio.channels), therefore  it is using SocketChannel objects rather than Sockets. Here, the SocketChannel is configured in blocking mode before being passed to the interceptor, which greatly simplifies its usage.

After adding a couple lines of code everywhere a socket or socket channel is accepted/connected and writing a very simple implementation of NetworkConnectionInterceptor where the client must send an int value '1' and the server must respond with '2', I got it working between nodes and driver and between clients and driver. I've attached the source of the interceptor mechanism if you want to take a look at it.

So this proves the concept is working and that's the gist of what I intend to deliver in 5.2.

I have registered a feature request to track it: JPPF-453 Network communication interceptors

I'd be using a different protocol than jmxmp, a custom one which includes auth.

For now, I'll stiick with JMXMP. It does have built-in auth features, in particular via its SASL profile. It's just that JPPF isn't using it yet. It also works nicely with VisualVM (just drop the jar in VISUALVM_HOME/platform/lib) and uses nice and simple service URLs.

In the hope that it clarifies things a bit, the version of JMXMP used in JPPF is a fork from the OpenDMK project, which appears to have been dead in the water for several years now. It's been modified in order to modernize it to Java 7 features (it was designed to work with JDK 1.3) and to make it more scalable, in particular by making it use a shared thread pool based on the java.util.concurrent APIs (the original version is creates a crazy number of threads). Additionally, the project's licensing allows it to be re-licensed under the ASL 2.0, as long as it's clearly stated. All this to say that we have full control over the source code and we can modify it as needed. Historically, JMXMP (and the generic connector it is part of) used to be called the reference implementation for JSR 160.

I've also looked at the links your provided for JBoss JMX remoting, and I see two blockers for its use in JPPF:
- it's licensed under the LGPL 2.1 and from my experience it doesn't with the ASL 2.0 JPPF is lecensed under
- since it relies on HTTP for communication, it requires an HTTP server for each node or driver that starts a JMX remote connector server

Sincerely,
-Laurent

[brian]

Brilliant! The interceptor API looks good, will give it a try when available in some release (beta is fine too).

I may be able to make do with pluggable sasl provider in jppf's jmxmp, I will run some tests in next few days and get back to you.

[brian]

Hi,

I investigated use of custom SASL provider with JMXMP and the problem I see if getting visualvm/jconsole client to use a custom SASL provider. For example I can start visualvm like this:
jvisualvm --cp:a jppf-jmxremote_optional-1.1.1.jar
And support for jmxmp is discovered via JMXConnectorProvider,JMXConnectorServiceProvider files in META-INF/services. But from what I can tell to have a JMX client use a custom SASL provider you need to call Security.addProvider and env.put("jmx.remote.profiles", XX) and how is that possible for the 3rd party visualvm/jconsole apps without forking/modifying their code? Perhaps there is a plugin point in those tools for SASL that I'm not aware of.

On the other hand getting visualvm to understand a custom protocol is easy, we just include another jar again with JMXConnectorProvider,JMXConnectorServiceProvider files in META-INF/services. So if jppf supported pluggable JMX connector providers too then the authenticated JMX story is good.

I also considered adding authorization to JMX. Our approach here is to define a MBeanServerForwarder which wraps all MBeanServer invocations with permission checks, and install that forwarder via javax.management.remote.JMXConnectorServer#setMBeanServerForwarder before it is started. Looking at org.jppf.management.JMXMPServer we'd just need a way to call setMBeanServerForwarder before the server is started.

Thanks,
Brian

[lolo]

Hi Brian,

Regarding the SASL functionality, you raised two concerns that I believe can be addressed easily:

you need to call Security.addProvider

This is one of two ways to add a provider. The other is to declare it in JRE_HOME/lib/security/java.security, as specified here. In fact, in the example code for JMXMP the call to Security.addProvider() is no longer necessary, because the provider "com.sun.security.sasl.Provider" is already declared in the java.security file for Java 7 and Java 8.

and env.put("jmx.remote.profiles", XX)

True. Not only that, but there are other environment properties, such as "jmx.remote.sasl.callback.handler", whose value is a complex type, rather than a string or a number, so it can't be set as a system propery either. What I can propose is to have a pluggable JMXClientEnvironmentProvider and JMXServerEnvironmentProvider that would provide environment properties to add to or override those passed to JMXConnectorFactory.newJMXConnector() and JMXConnectorServerFactory.newJMXConnectorServer(). It's a relatively simple modification of the JMXMP connector code and it would cover the SASL use case and probably others.

Our approach here is to define a MBeanServerForwarder

Sure, that shouldn't be a problem. I can even create a default forwarder where all methods are delegated to the MBeanServer, so you only override those you need to change. It could be plugged via a configuration property, e.g. "jppf.management.server.forwarder = my.forwarder.Implementation". I actually registered the feature request JPPF-456 Pluggable MBeanServerForwarder for drivers and nodes for this.

So if jppf supported pluggable JMX connector providers too then the authenticated JMX story is good.

Currently that's a no go for me. JMXMP (or rather the generic connector it is an implementation of) has the particularty of providing a customizable ObjectWrapping. This allows JPPF to use its pluggable serialization scheme for MBean methods' parameters and return value. As far as I can tell this is the only JMX remote implementation that allows it.

Sincerely,
-Laurent

[brian]

Thanks for the response!

With our central JDK installs I won't be able to add a custom provider to java.security file, the installs are out of my control.

Your suggestion about modifying jppf to support setting environment properties makes sense. But does it work on the visualvm/jconsole side too? Without modifying those third party sources I don't see how to plug in the same sasl customization to jmxmp on the tool side.

Good idea on the forwarder. Although I may need to instantiate my own forwarder impl in order to inject the network resource it requires to do permission lookups, so the config property may not be sufficient.

I understand about the jmxmp dependency, that makes sense thanks for explaining.

[lolo]

Hi,

Ok, I understand you're not allowed to modify the JVM setup, that makes sense. There are still 2 possibilities:
- if you use the existing SASL provider, then it is already declared in java.security and you don't need to call Security.addProvider()
- if you use a custom provider, then you can call Security.addProvider() from within a pluggable forwarder's constructor. I can guarantee (and document) that the forwarder is intantiated at an approprioate time for that.

But does it work on the visualvm/jconsole side too

Yes, as long as you can add the jar with your implementation of JMXClientEnvironmentProvider to the tool's classpath, just like you do for the jmxremote_optional jar file. The environment provider will be used every time a JMXMP connection is attempted, no matter where it is attempted from, because it is triggered from the JMXMP code, not from the tool's code.

Edit: I actually gave it a try, using an environment provider implementation that just shows an information dialog with JOptionPane, and now I can see this annoying popup dialog every time I start VisualVM

I may need to instantiate my own forwarder impl in order to inject the network resource it requires to do permission lookups, so the config property may not be sufficient

I do not understand. Do you have a use case or example that is not covered by the feature request?

Thanks,
-Laurent

[brian]

you can call Security.addProvider() from within a pluggable forwarder's constructor

I wonder why you suggest calling addProvider in MBeanServerForwarder ctor instead of in the environment provider. Visualvm will also need addProvider but there will be no MBeanServerForwarder on visualvm side to invoke it. Whereas the environment provider would be on both sides so seems more appropriate place to call it, no?

as long as you can add the jar with your implementation of JMXClientEnvironmentProvider to the tool's classpath

This sounds good. However I cannot confirm/test because these classes JMXClientEnvironmentProvider/JMXServerEnvironmentProvider don't seem to exist, I guess they are only hypothetical at this point? Perhaps if you sent me instruction to replicate the annoying visualvm popup on my end it would be more clear what you suggest.

Do you have a use case or example that is not covered by the feature request?

Perhaps. The forwarder impl I'd plug in would use Spring Security AccessDecisionManager[1] to authorize each action. Creating my AccessDecisionManager takes some work because it will go over the network to do permission checks. It requires a hostport and some other config. If you instantiate my forwarder using reflection, who will supply it the hostport? It could find the hostport using static method like System.getProperty however we do not typically configure our apps with system properties. I'm wondering if you need some programmatic way of injecting a forwarder. jppf.setForwarder(my_already_constructed_impl)

[1] http://docs.spring.io/autorepo/docs/spring-security/current/apidocs/org/springframework/security/access/AccessDecisionManager.html

[lolo]

Hi,

Regarding your first point, I also wonder why I suggested the MBeanServerForwarder, since it's only applicable to the server side of a remote JMX connection. Thanks for catching that.

if you sent me instruction to replicate the annoying visualvm popup on my end it would be more clear what you suggest.

Sure. I've put together a zip file that contains the modified jppf-jmxremote_optional jar and the source for an implementation of the ClientEnvironmentProvider interface. Instructions are provided in the readme.txt. You can get the zip file here: http://www.jppf.org/private/EnvironmentProvider.zip.
I have also committed the related changes to the jmxremote source: https://sourceforge.net/p/jppf-project/code/4074/

It requires a hostport and some other config. If you instantiate my forwarder using reflection, who will supply it the hostport?

Configuration parameters could be supplied in the same configurationa property that defines the forwarder's implementation class name, for example:

Code: [Select]

jppf.management.server.forwarder = my.forwarder.Implementation param1 ... paramNThe code that instantiates the forwarder could then look for a constructor that takes a String[] and use it to pass the parameters, and if no such constructor exists, if could then look for a setParameters(String[]) method and invoke it. We've done that for other config properties and it works great.

-Laurent

[brian]

Thanks Laurent, I understand your environment provider now. I will attempt integrating it with my SASL provider and let you know how that goes. I don't see any reason it wouldn't work at this stage. I may not have a chance to attempt this this week.

Regarding the forwarder, that's a very nice technique for passing config to a reflection-instantiated object, I'll remember that one! Although I don't think it will work for my case, looking closer at the code my forwarder also requires references to shared objects like a connection pool. (In case you are wondering where these shared objects will come from, I plan to have all jppf components embedded within a running JVM using my own main class, ie never using jppf's startX.sh scripts.) If it's important that jppf construct the forwarder, perhaps there could be a way to programmatically fetch a reference to the forwarder after construction/installation, at which time I could do a late injection of required dependencies?

[lolo]

Hello,

perhaps there could be a way to programmatically fetch a reference to the forwarder after construction/installation, at which time I could do a late injection of required dependencies?

Sure, I could add an API to obtain a reference to the forwarder.
What I propose:
- add a method getMBeanServerForwarder() to the JMXServer interface
- add a method getJMXServer(boolean secure) to the  JPPFDriver class (both secure and non-secure JMX servers can coexist)
- this should allow you to get the forwarder with:

Code: [Select]

MBeanServerForwarder myForwarder = JPPFDriver.getInstance().getJMXServer(false).getMBeanServerForwarder();Will this be ok?

Also, regarding the other features we discussed in this thread, I have already implemented and published the documentation for:
- network interceptors, with a sample here
- JMX environment providers
Any comment will be appreciated

Thanks,
-Laurent

[brian]

I think this all came out very nicely Laurent.

Will this be ok?

Yes. Would you also add an API to obtain a reference to the network interceptor? I need to inject the same authz strategy object to the interceptor as I do the forwarder. FWIW I only need to do this on the server (i.e. onAccept) side of each connection.

[lolo]

Would you also add an API to obtain a reference to the network interceptor?

Done.

-Laurent

[brian]

Interesting, an interface field.

- Is the list threadsafe to iterate? Maybe there is a general statement on jppf API threadsafety which covers this that I'm not aware of?

- When client+driver running in same JVM how many elements would be in the list?

[lolo]

Since the list is unmodifiable, it can be safely read by as many threads as you wish.
The number of elements in the list depends only on the number of service definitions in the classpath, where a service definition is a non-comment line in any META-INF/services/org.jppf.comm.interceptor.NetworkConnectionInterceptor file found in the classpath.

I hope this clarifies,
-Laurent

[brian]

Since the list is unmodifiable, it can be safely read by as many threads as you wish.

Ok. Just to be clear though in general that is not a safe assumption to make in Java, it may be an unmodifiable view over a mutable list which another thread is mutating. In your case we're fine because its unmodifiable and immutable.

Minor feedback:
- It might make sense for AbstractNetworkConnectionInterceptor to keep the Socket overloads abstract and just implement the SocketChannel overloads with onX(channel.socket()), rather than having both sets of methods call new IS/OS methods. Just seems like a simpler abstract impl and .socket() should be safe to use given you set blocking mode?

- Docs say network interceptors are used for all types of connections including JMX, is that right? When I run vanilla admin UI against driver with an interceptor I don't see onAccept/onConnect even getting called in driver, whereas a vanilla node trying to connect to driver does cause onAccept to be called. However, admin UI doesn't show driver when interceptor installed, so something is blocking it..

[lolo]

Hello,

In your case we're fine because its unmodifiable and immutable.

Yes, you are absolutely right.

It might make sense for AbstractNetworkConnectionInterceptor to keep the Socket overloads abstract and just implement the SocketChannel overloads with onX(channel.socket())

Indeed. I was wrongly assuming that operations on the streams obtained from channel.socket() would raise exceptions, but after looking at the related javadoc, it is now clear that exceptions are raised only when the channel is in non-blocking mode. I'll do the change, this will simplify both the code and the documentation, thank you.

I don't see onAccept/onConnect even getting called in driver

They are called, but there is no trace in the interceptor handling code. Furthermore, I added additonal output in the interceptor sample for testing, and I could see that on the driver side, the onAccept() calls are simply stuck trying to read a response that the client/admin tool never sends. Because of that, it never reaches the point where the sample prints to the output console.
I will try and modify the sample to use an SO_TIMEOUT on the channel/socket in order to fail the authentication if a read times out.

I've committed an updated version of the interceptor code so that debug-level logging is performed whenever onAccept() / onConnect(), it will be part of the next nightly build (around 9:15 pm UTC).
To see the new traces, add the following in the log4j config file:

Code: [Select]

log4j.logger.org.jppf.comm.interceptor=DEBUG
Sincerely,
-Laurent